This document defines goals and tasks for each sprint of the Teiid Data Services WebUI Development. Features and bugs should be tracked in the JIRA TeiidTools project.
- Raw JSON results
Sprint 10
Jan 9th - Jan 20th, 2017
The sprint will concentrate on bug fixing and enhancements.
1. DataService Editor
- Table - Column display enhancements [TEIIDTOOLS-58]
- Join Criteria enhancements [TEIIDTOOLS-59]
- Temp VDB management [TEIIDTOOLS-60]
- Exclude inactive sources from source list [TEIIDTOOLS-62]
2. Context-sensitive Help
- Complete context-sensitive help [TEIIDTOOLS-34]
- Complete i18n framework [TEIIDTOOLS-35]
3. DataSources
- Verify JDBC Filtering capabilities [TEIIDTOOLS-57]
- Add ability to view the tables / columns for a data source (tabular or tree view in addition or instead of the DDL) [TEIIDTOOLS-61]
- Select 1 or 2 sources on Data Source Summary page and proceed directly to DataService creation [TEIIDTOOLS-55]
- When configuring a New Data Source.. add check-box to proceed directly to DataService creation [ ]
4. Miscellaneous
- Correct / improve the information shown for each row on the Data Service Summary page [TEIIDTOOLS-63]
- Remove ListView toggle button from summary pages. (We don't allow listview / cardview toggling) [ ]
- Repository Structure Improvements [TEIIDTOOLS-40]
- Update Save button on Git config prefs [TEIIDTOOLS-76]
- General issues with preferences [TEIIDTOOLS-64]
- Prompt to save changes on preferences page [TEIIDTOOLS-64]
Sprint 9
Dec 3rd - Dec 21st, 2016
1. DataService editing for multi-source cases (MARK)
Add capability to create / edit Data Services which can combine multiple sources
- Single Source / Multi-source selector determines page content.
- Multi-source is limited to 2 sources (similar to OpenShift v2 wizard)
- Note that with Teiid JDBC connections to VDBs, we will probably need to filter out SYS, SYSADMIN and pg_catalog schema
- Include VIEW as allowable Table Type in addition to TABLE
- related to: TEIIDTOOLS-39
2. Context-sensitive Help [TEIIDTOOLS-34]
- Complete the pageSlider content for each page (TED, DAN)
- Add field level help where appropriate (TED, DAN)
3. Docker Images
It was recommended after the last demo to create docker images containing dbs with well defined data for testing and demos.
4. Test and improve JDBC connection handling in KomodoTeiidService
- The method of obtaining JDBC connection for JDBC sources needs to be tested (Does it work with wildfly? and can it be improved?)
- Current implementation waits for the default timeout (2mins?) when attempting to obtain the JDBC connection. Maybe need a configurable timeout.
5. Misc Improvements
- Data Service Test page - Add a raw Json tab for the OData results in addition to the tabular results. (PAUL)
- Data Source Summary page - Ability to view the tables / columns for a data source (tabular or tree view in addition or instead of the DDL?) (MARK)
- Resizing (narrowing) browser window results in left-side vertical navigation labels to extend into main panel. Be nice to set minimum width (
- Fix DS Builder main bar graphic to show full name. Currently truncated to "Data Service Build"
- When configuring a New Data Source.. add check-box to proceed directly to New Data Service page (MARK)
- Add About version information for DSBuilder and Komodo engine (PAUL) [TEIIDTOOLS-48]
6. Make editing of imported and bottom-up DataServices consistent
Editing of imported DataServices vs bottom-up DataServices is not consistent. (The assumption that source connections must exist is part of the reason). Here are a couple scenarios:
a. Import one of the DataService samples. Immediately editing the dataservice will not show any sources. After import the source has not yet been deploy.
b. Create a DataService from the bottom up. Export of the service does not yet include the source vdb(s) and connections...
see: TEIIDTOOLS-41
Sprint 8
Nov 14th - Dec 2nd, 2016
1. Source Schema selection (JDBC sources) [TEIIDTOOLS-39]
The main priority for this sprint is to incorporate schema selection for JDBC sources. This will allow the user to narrow the source schema to include only the portion of interest.
- Wireframe the page design for jdbc sources
- Consider how future non-jdbc sources may fit into the design.
- Implement JDBC schema selection (possibly a minimal implementation to start)
2. i18n Support [TEIIDTOOLS-35]
Continue implementation of translations for the remaining pages and controllers.
3. Context-sensitive Help [TEIIDTOOLS-34]
- Rather than provide detailed/verbose content throughout the Web UI to explain actions, panels, etc.... other tools have used a context-sensitive help panel
- So when a user creating a new Source or Data Service... the Help panel would show a description of what those objects are and maybe the next steps.. plus links to other relevant "Help topics"
- Example directives that might work are found here: AngularJS Pageslide directive demo
- PatternFly field level help guidance: Field Level Help | PatternFly
- PatternFly tooltip guidance:
4. Make editing of imported and bottom-up DataServices consistent
Editing of imported DataServices vs bottom-up DataServices is not consistent. (The assumption that source connections must exist is part of the reason). Here are a couple scenarios:
a. Import one of the DataService samples. Immediately editing the dataservice will not show any sources. After import the source has not yet been deploy.
b. Create a DataService from the bottom up. Export of the service does not yet include the source vdb(s) and connections...
5. DataService editing for multi-source cases
Propose deferral of the implementation until next sprint, but start design and wireframe mockups in sprint 8.
Sprint 7
October 31 - October Nov 11, 2016
UI Framework
1. Context-sensitive Help panel [TEIIDTOOLS-34]
- Rather than provide detailed/verbose content throughout the Web UI to explain actions, panels, etc.... other tools have used a context-sensitive help panel
- So when a user creating a new Source or Data Service... the Help panel would show a description of what those objects are and maybe the next steps.. plus links to other relevant "Help topics"
- Example directives that might work are found here: AngularJS Pageslide directive demo
- PatternFly field level help guidance: Field Level Help | PatternFly
- PatternFly tooltip guidance:
2. i18n Support [TEIIDTOOLS-35]
DataService Testing
1. Testing improvments / OData queries [TEIIDTOOLS-36]
Source Editing and Deployment
1. Source Status detection and update [TEIIDTOOLS-37]
2. Translator selection strategy [TEIIDTOOLS-38]
3. JDBC schema selection [TEIIDTOOLS-39]
Repository
1. Structure improvements [TEIIDTOOLS-40]
Sprint 6
October 17 - October 28, 2016
Proposed work for sprint 6 is to solicit feedback on the current UI and continue to solidify the current functionality
1. Source editing and deployment
- Source Status: Currently the source status can be 'New' or 'Active'. If the source is deployed successfully, the status goes to active. More work is required to accurately show the different states. For example, it may take awhile for the source metadata to load upon deployment. This should be reflected in a status of 'loading', and the user should not be able to view or use the source until it has finished loading.
- Source import properties: When a new service source is created or updated, currently we are adding a few import properties. Eventually we need to expose these properties to the user (maybe a wizard). And also allow the user to select the desired schema for the source. This is probably a task for the next sprint.
- Maybe hard-code Table Types = TABLE for now
- Push rest to Sprint 7/8
- Translator selection. We should be able to select the necessary translator based on the connection type. Hopefully eliminate the translator dropdown altogether - hide it and make it an 'advanced' option to change the translator?
- When we create a DS/RA in Designer using Teiid API, we provide the "type" (or translator). You'd think they could give it back via the API
2. Dataservice editing and deployment (Mark)
- Add connections and drivers on create. Currently when a dataservice is created, the connection and driver is not added to the dataservice.
- Need to determine the connection and driver used and add them to the dataservice repo object. (derived from the Data Source object from Teiid. See Teiid Connection Importer)
- Editing improvements.
- When editing the dataservice, the referenced source / table are not pre-selected on the edit page. This needs to be implemented.
- Round trip editing.
- Need more testing and improvements to round trip editing of services. Make sure the imported services can be edited, etc.
- Service View generation. The current view generation code could use improvement. The generated view currently creates a unique rowid column to satisfy requirements for successful odata deployment. It apparently uses a function ROW_NUMBER that is not compatible with all source types (eg sqlserver). One possible improvement - If the source table has a PK or UC, should rely on it instead of trying to create the unique column.
3. Dataservice test page improvements
- OData link - rather than take the user to a new browser tab to show the results, just embed the result display in the test page.
- JDBC ad hoc queries - should we keep the ad hoc query capability currently on the page, or just use the OData endpoint for building and submitting queries?
- OData querybuilder - do a browser search on 'odata query builder'. Looks like there are some examples or existing stuff that we may be able to leverage if we decide to go this route.
- Fixes for query components - if we decide to keep the query components currently on the page, they need some debug. Current Issues - 1) need to click in query area to see the query 2) result table header and columns initially not lined up (page resize fixes it)
- Do not allow query until dataservice is finished loading. Currently, the user can submit a query even if the dataservice is still loading - in which case the query fails. Need to detect dataservice is loading and not allow query submittal.
- Detect if user has set the jdbc user/pwd. A common problem on this page is to submit a query before setting the teiid user/pwd preferences. Maybe we can detect if these have been set first?
4. Framework improvements
- Currently most or all of the workspace rest services assume that the created / edited objects are at the workspace root. We should consider enforcing a workspace structure. For example, all of a users vdbs go under a /vdbs node - and so on. This will help prevent name collisions, etc.
- The Data Service artifact structure may work.
- Initial DSB will be defining single DS's with single VDBs containing a View model and 1 or 2 Source models.
- FYI.... Source VDBs are somewhat temporary. Once a DS is created the Source VDB model DDL is injected into the DS vdb. Deleting the Source will not affect the DS vdb contents.
Sprint 5
Completed Oct 14, 2016
Sprint 4 was delayed so we were also delayed in the sprint 5 release. Here are the highlights of tasks completed in Sprint 5.
1. UI navigation
- The navigation for the entire application was reworked. It is now more consistent with other jboss web apps under development
- LH navigation bar for Home, DataServices and Sources
- Breadcrumbs within each subsection
- Top title bar was replaced with DataService Builder logo
2. Dataservices Summary page
- Summary rows were improved to show the constituent parts of the service
3. Source editing improvements
- round trip editing of the sources was completed.
- added importer properties for sources
4. Dataservice editing
- Improved the Dataservice create and edit pages. The user can now define a dataservice from the bottom up, using available sources. Currently the user is limited to selection of a single source table, but this will be expanded in future sprints.
5. Dataservice testing
- Minor fixes on the Dataservice testing page.
6. Framework improvements
- Various improvements in teiid-modeshape and komodo frameworks.
Sprint 4
August 8 - August 26, 2016
- Develop a user login strategy
- Workspace definition for a restricted user's workspace.
- user is assigned a specific workspace location for their dataservices or the dataservice is somehow keyed to a user.
- Changes to WorkspaceManager to allow specified workspace location? Pass in user folder for root?
- How to prevent user2 from seeing user1's Dataservices?
- Consider allow user1 to access a shared ConnectionDefinition/Datasource location in the repo?
- Securing REST operations
- Identifiying operations to a specific user and allowing/disallowing them access
- Initial authentication to REST service and returns of security identifier to client that can be cached and used by client to service future REST operations
- Client facility to initiate Login/Logout
- Integration with keycloak authentication/authorisation service to utilise users/roles services offered by system
- Documentation of security systems
- Installation and configuration, covering initial setting up of users/roles
- Design of security sub-systems to promote confidence in application and its processes
- Workspace definition for a restricted user's workspace.
- Improve UI editing capabilities
- Develop strategy and states for object status (repo vs server deployed)
- (VDB / Datasource / Connection DS) pages
- Mock up and implement pages for editing
- Dataservice page additions / improvements
- Designate the ViewModel / View that represents the Dataservice
- Add capability to test the service from edit page
- Add capability to edit the service view (ddl box initially?)
- Add capability to add / remove source vdbs.
- Connections page additions / improvements
- Complete the property editing capability
- Incorporate sequencers
- Complete teiid-modeshape sequencer work for dataservice and datasource
- Convert komodo codebase to use the sequencers (import / export / test cases / etc)
Sprint 3
July 18 - August 5, 2016
- Enable Export of a Data Service to a GIT repository
- Preference page for setting the git repository
- Action in dataservice builder to 'publish' a dataservice to git
- framework updates to allow export to git
- Incorporate Dataservice sequencer
- Pull the teiid-modeshape sequencer work into komodo codebase.
- Replace the functionality in DataserviceConveyor.dsImport() with the requisite calls necessary to activate the sequencer
- Update TestUtilities test dataservice zips to the new format expected by the sequencer and ensure all unit tests succeed
- Incorporate Patternfly directives for the summary pages
- connect patternfly toolbar directive to real dataservice data
- include actions from current dataservice summary page
- Evaluate competitor products
- Compare UI flow with similar tools
- Evaluate application navigation
- Evaluate use of breadcrumbs (or other) for easy navigation.
- Improve Dataservice editing capability
- The 'edit' page for a dataservice is currently in place but only allows the user to change the name and description of a Dataservice.
- The edit page should allow the user to
- Designate the ViewModel / View that represents the Dataservice
- Initially a 'Dataservice' is a single view but could potentially be expanded as users want to have multiple views
- Define the Dataservice View.
- Initially a simple text area, but later replaced with a graphical editor.
- Test the Dataservice. Potentially add a button to test from this page.
- (stretch goal) Add/Remove available 'Datasource' Vdbs from the repository to be used in the Dataservice.
- Adding a vdb would (1) add the source vdb (2) add the connection definition it uses (3) add the drivers if necessary
- Designate the ViewModel / View that represents the Dataservice
- Develop Connection/ConnectionDefinitions management page
- The connections page will show the Datasource objects in the komodo repo.
- Terminology : In the UI, a "connection definition" is our komodo datasource object - connection properties for a jboss datasource
- The summary page will look similar to the Dataservices summary page (patternfly toolbar directive)
- Allow user to import / export their own definitions.
- Allow user to create their own definitions from scratch, including upload of drivers
- The 'play/test' button would deploy to jboss AS to test it (ping to ensure validity).
Sprint 2a (mini-sprint 1 week)
July 11 - July 15, 2016
Goals
- Odata tie-in
- Merge the PR for resteasy downgrading. This will mean that a new resteasy install is no longer required on top of jboss/eap
- Check that an odata endpoint can be successfully constructed and called by the teiid runtime for the example dataservice
- Implement a notional UI to display the odata endpoint and consider whether it could also be tested on-the-fly?
- Update the user installation instructions to remove any reference to resteasy upgrade and include instructions on installing only wars into existing jboss/eap
- Teiid Credentials
- Implement a preference view, allowing modification of teiid instance credentials so users need not access swagger interface.
- Update the user installation instructions to include the new preference view
- Building
- The jenkins build should be automatically building and churning out builds ready for snapshot release.
- Ensure that the build is working correctly, not producing test failures and a building is ready to be published by the end of the sprint.
Sprint 2
June 15 - June 29, 2016 July 8th 2016
Goals
- Define the constituents of a DataService
- Provide tools (rudimentary) for the creation of data sources, translators, ....
- see: The specified item was not found.
- Ensure a generic tool that covers all possibilities then specialise for individual types, eg. a basic datasource wizard contains a single table of teiid properties
- Provide tools (rudimentary) for the creation of data sources, translators, ....
- Develop ability to deploy a DataService
- REST service to deploy a DataService to teiid instance.
- Extract the dataservice-vdb.xml from the repository DataService
- Deploy it to the teiid instance
- NOTE: This sprint will use a handcoded 'DataService' with a valid view definition / sources that have been pre-deployed to the server.
- Datasources consist of at least 2 components, the driver and the configuration properties. Both must be present at runtime for a datasource to work effectively
- REST service to deploy a DataService to teiid instance.
Stretch goals:
- REST service to run query against a deployed DataService and return results
- Develop 'test' page to show the deployed dataservice and specify/submit ad-hoc queries
Sprint 1
May 20 - June 3, 2016 - June 13, 2016
Goals
- Display a 'DataServices' page, showing a user's DataServices within the komodo repo.
- initially the userName can be hardcoded. (later will be hooked up to login). (Due to significant code changes for user home directories - pushing to Sprint 2)
- Allow export of a DataService to the file system (aService.zip). zip file will initially include the dataservice dynamic vdb.xml
- Allow import of a DataService from the file system (aService.zip).
- Allow create, delete, clone of a DataService in the users komodo repo.
- All operations update the displayed DataServices page
Stretch goals:
- Allow export of a DataService to a git repo location.
- Allow import of a DataService from a git repo location.
Sample Layout:
(tabular layout is shown, or could be card layout)
Task breakdown:
1) Komodo repo structure changes
- allow folder structure under the workspace (folder nodes). This enables arbitrary structures like /workspace/myOrganization/mdrilling to define user folders (supporting folder structures will be implemented but significant code changes required for implementing user home directories - pushing to Sprint 2)
- create a DataService node. DataServices can exist under the workspace or under a folder. Initially a DataService contains one child - a VDB (the 'data service' vdb).
- Update komodo cnd to include DataService
- Data Service containment (TBD after Sprint 1)
- initially a VDB with name save as Data Service
- connection & data source information sufficient to deploy/create data source/RA on the server and wrapper source/view definitions (DDL)
- ???
- Data Service containment (TBD after Sprint 1)
- Update KomodoLexicon to include DataService
- Update Repository interface and implementation with "home directory" API (Due to significant code changes for user home directories - pushing to Sprint 2)
- Modify relational classes to add 'stuff' to the home directory rather than the workspace (Due to significant code changes for user home directories - pushing to Sprint 2)
- Central singleton to fetch the user and home directory from? (Otherwise username needs to be included in all API methods!)
- How does such a singleton distinguish between user1 and user2?
- Username will be hardcoded to start with but could get limited very quickly to singleton if not consider multi-user case
- (Due to significant code changes for user home directories - pushing to Sprint 2)
- Update komodo cnd to include DataService
2) Storage framework plugin for filesystem - DONE
- The storage framework contains a service for git but will need one for the filesystem. This would be the default plugin if no other available
- Storage connectors require a set of Properties that provide the settings for how the connectors operate, eg. git plugin needs a URI for the remote git repo.
- Where would these preferences be stored?? Given a user, can we fetch these properties from the environment?
- Workflow for file import:
- User displays client file dialog to choose a file. (Consider angular security implications for this)
- File data is posted to the rest service as an application/octet stream(?) or multipart/form-data
- Data is recreated in server tmp (necessary?) and imported into the workspace using VdbImporter
- This workflow precludes the use of a framework plugin??
- Browse -> seems unlikely to browse the server's filesystem
- Read -> since browse is unlikely then so is this since no idea what filesystem location to read from
- Write -> writing a file would send it to the filesystem BUT not much use unless it goes back to the client, presumably through the rest services
- Filesystem storage is a service but will look very different to the paradigms of git and jdbc
3) Develop web pages
- DataServices summary page (table or card layout)
- Empty DataServices page (if user does not have Data Services, a 'no services' image is shown with a button to create a Data Service.
- New DataService page (enter a DataService name and description)
- Clone DataService page (enter a new DataService name and description)
- Import DataService page? (selects a file from file system for upload)
- Interrogates Rest Service for installed storage plugins and asks for what parameters are required then displays them as input boxes to be filled in, eg. git URI
- User selects their preferred type of storage, eg. file, git
- Need to store this preference in user preferences?
- Preference used for import/export through pages so Rest Call to fetch this preference?
4) Develop REST methods required for Data Services page (examples below)
- URIs need to be modified since Vdbs exist under workspace atm
- Whether user home directory should appear in the Rest URI or as a parameter, eg. /workspace/vdbs?user=bob --> /workspace/myOrg/bob/vdbs
- RestUriBuilder must handle both User and DataService
- getDataServices( repoPath ). Get the DataServices for the specified repo path. Navigate to the Komodo repo location "/workspace/orgName/userName" and returns the DataServices object details.
- addDataService( repoPath, serviceName ). Add a new DataService object with the name 'serviceName' at the repoPath location.
- deleteDataService( repoPath, serviceName ). Delete the DataService with the name 'serviceName' at the repoPath location.
- cloneDataService( repoPath, serviceName, copyServiceName ). Copy a DataService 'serviceName', creating a new DataService 'copyServiceName'.
- importDataServiceFromFile( ). Import a DataService from file system. 'aService.zip' - zip file format
- exportDataServiceToFile( ). Export a DataService to a file ('aService.zip').
- importDataServiceFromGit( ). Import a DataService from a git repo location. (stretch goal)
- exportDataServiceToGit( ). Export a DataService to a git repo. (stretch goal)
- importDataService(). Import a DataService. Expects an http parameter containing the type of storage
- Import of the data service will import the data service zip file. Initially the dynamic vdb xml within the zip file will be used to create the data service in the repo.
- exportDataService(). Export a DataService. Expects an http parameter containing the type of storage
- Rather than specific methods naming the storage location, use importDataService() and provide an http parameter with the requested type, eg. "git", "file".
- The plugin storage framework uses this type to find a StorageService that has such a type and provides a StorageConnector that takes parameters to do the import/export work
- Export of the data service will be a zip file. The zip file will initially include the dataservice dynamic vdb.xml. In future sprints we will add dataSource configuration files or whatever is required.
Comments