(ORG-562) Automated Project Matrix - External/Awestruct sites

[aslak]

There is work being done to automate the project matrix page on jboss.org/projects (ORG-562) based on metadata from Magnolia. There is another feature request to add descriptions for each project to be included on the page (ORG-1408).

According to Mark Newton in an internal mail, there will be some 'properties' files for Awestruct based sites.

Where can I find some information on this file, format/ content/ location?

As more and more projects move to Awestruct, won't having a manual properties file for these sites just slowly move this to a manually maintained page again?

In the Arquillian project we've been look at exposing APIs on our site as well as the traditional html based view. Meaning we will be creating 'static' json/xml/yml APIs for e.g. Guide/Documentation API (tags, links, internal, external), Container/Extension API (maven artifacts, versions, configuration options). Similar we could create a Project API which would contain information about where source code is located, build server, documentation site etc etc..

Is this something we could integrate in the Project Matrix automation job instead of harcoding properties files?

[aslak]

Just to cleaify:

So instead of having a arquillian-project.properties file located somewhere on jboss.org, you have the url: http://arquillian.org/api/project.json

You probably don't need to call that API pr user request as this is mostly static information, but you could regen the matrix page e.g. once a week with fresh information.

Then the content of the Matrix Project Page is in the hands of the people that actually control the info, and it's just a part of the normal project workflow. No content duplication.

[lightguard]

Maybe I'm missing something, but couldn't we use the projects in JIRA for this? Certainly you'd need to do some trimming and condensing of projects, but all of our projects use the JBoss JIRA don't they?

[dan.j.allen]

I tried to use JIRA for this originally, based on advice from Aslak. It gets you part of the way there, but not all the way. To flip it around, JIRA is a great input source, but it's not the only input source.

Think of the architecture this way. We are creating a project catalog pipe (ala Yahoo Pipes). It will have multiple inputs. It may produce multiple outputs. One of those outputs is jboss.org/projects. Another is a REST endpoint for the catalog.

Btw, I did create a script to pull from JIRA:

https://gist.github.com/2699890

which originally seeded this list:

https://gist.github.com/7bfe5d4660d201463952

Obviously, there is still work to do. I think the step to iron out right now is the API and data structures. Then we'll be ready to begin automation.

[lightguard]

We really don't have a single source of truth for this information? That honestly surprises me.

[dan.j.allen]

Nope, we don't. It surprised me too at first, but when I thought about it, I never had a place to enter the information, so how could there be.

JIRA is also not sufficient because it's not necessarily the case that there is one JIRA per project, or that the project even uses JIRA. I don't think it's a good idea to tie metadata to a system like JIRA or Magnolia. This information should be in a technology agnostic location (just a data file). The project is free to generate this file in whatever way they want. We are going to include ours in the Arquillian website, likely. The central catalog then only has to maintain a list of URLs pointing to the data files.

[lightguard]

Honestly seems like some sort of script that should be run when we create a project (not that we do this all the time). The script would allow JIRA to be setup, add this information, maybe some other stuff as well. When everything was hosted like VCS then that would have been setup too. Maybe it's not such a good idea anymore. A simple datafile works just as well.

[dan.j.allen]

I'll copy here what I wrote in ORG-1408.

We've been talking for a while about automating the list of projects on the index page. As it is now, the list is deficient on several accounts. There are projects missing, projects which shouldn't be listed and entries with outdated links.

We can solve all these problems at once by pushing the project metadata (i.e., enlistment) out the projects and then using a script to aggregate the information together to generate the index page. This allows projects to maintain their own information and updates to be propagated automatically.

For project metadata, I'm thinking about using something like DOAP or shorthand versions of it. I've explored a few formats in which to store information. Here's are three examples of the JBoss Tools project:

DOAP

<Project xmlns="http://usefulinc.com/ns/doap#"
    xmlns:foaf="http://xmlns.com/foaf/0.1/"
    xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
    xmlns:jboss="http://api.jboss.org/doap-extensions">
    <name>JBoss Tools</name>
    <shortdesc>Eclipse-based tooling for developing with JBoss middleware</shortdesc>
    <description>An integrated collection of Eclipse plugins to support the development of applications...</description>
    <homepage rdf:resource="http://jboss.org/tools"/>
    <download-page rdf:resource="http://jboss.org/tools/download"/>
    <wiki rdf:resource="http://community.jboss.org/en/tools"/>
    <mailing-list rdf:resource="https://lists.jboss.org/mailman/listinfo/jbosstools-dev"/>
    <bug-database rdf:resource="http://issues.jboss.org/browse/JBIDE"/>
    <license rdf:resource="http://www.eclipse.org/legal/epl-v10.html"/>
    <maintainer>
        <foaf:Person>
            <foaf:name>Max Ryhahl Andersen</foaf:name>
            <foaf:homepage rdf:resource="http://community.jboss.org/people/maxandersen"/>
            <jboss:userid>maxandersen</jboss:userid>
        </foaf:Person>
    </maintainer>
    <repository>
        <GitRepository>
             <location rdf:resource="git://github.com/jbosstools/jbosstools-svn-mirror.git"/>
             <browse rdf:resource="http://github.com/jbosstools/jbosstools-svn-mirror"/>
        </GitRepository>
     </repository>
</Project>

JSON

{"Project":{ 
  "name": "JBoss Tools", 
  "shortdesc": "JBoss Tools", 
  "description": "An integrated collection of Eclipse plugins to support the development of applications...", 
  "status": "A",
  "download-page": "http://jboss.org/tools/download",     
  "mailing-list": "https://lists.jboss.org/mailman/listinfo/jbosstools-dev",     
  "homepage": "http://jboss.org/tools",     
  "GitRepository":{ 
    "location": "git://github.com/jbosstools/jbosstools-svn-mirror.git",         
    "browse": "http://github.com/jbosstools/jbosstools-svn-mirror"}, 
  "licenses":[ {
    "name": "Eclipse Public License v1.0",
    "url": "http://www.eclipse.org/legal/epl-v10.html"}],     
  "trackers":[ {
    "name": "Bugs",
    "location": "http://issues.jboss.org/browse/JBIDE"}],     
  "maintainers":[ {
    "name": "Max Ryhahl Andersen",
    "homepage": "http://community.jboss.org/people/maxandersen"} ], }}

YAML

- name: JBoss Tools
  shortdesc: Eclipse-based tooling for developing with JBoss middleware
  description: An integrated collection of Eclipse plugins to support the development of applications...
  status: Active
  homepage: http://jboss.org/tools
  download_page: http://jboss.org/tools/download
  wiki: http://community.jboss.org/en/tools
  mailing_lists:
    name: Development
    url: https://lists.jboss.org/mailman/listinfo/jbosstools-dev
  issue_trackers:
    name: Issues
    location: http://issues.jboss.org/browse/JBIDE
  license:
    name: Eclipse Public License v1.0
    url: http://www.eclipse.org/legal/epl-v10.html
  maintainers:
    name: Max Ryhahl Andersen
    homepage: http://community.jboss.org/people/maxandersen
    username: maxandersen
  GitRepository:
    location: git://github.com/jbosstools/jbosstools-svn-mirror.git
    browse: http://github.com/jbosstools/jbosstools-svn-mirror

I'm in favor of YAML. But we could allow projects to choose the format they prefer, I suppose.

[aslak]

The problem with the initial project setup is that this information change over time, even tho not often. So we're back at maintaining it manually again.

A lot of projects recently moved from SVN to Git. Some projects are moving from Magnolia to Awestruct. Projects change names.

[dan.j.allen]

Btw, I got these ideas from the SourceForge API Wiki.

[dan.j.allen]

Aslak Knutsen wrote:

Just to cleaify:

So instead of having a arquillian-project.properties file located somewhere on jboss.org, you have the url: http://arquillian.org/api/project.json

You probably don't need to call that API pr user request as this is mostly static information, but you could regen the matrix page e.g. once a week with fresh information.

Then the content of the Matrix Project Page is in the hands of the people that actually control the info, and it's just a part of the normal project workflow. No content duplication.

+1

[aslak]

Apache use DOAP as well: http://projects.apache.org/

[dan.j.allen]

Btw, the reason I created the list of projects is:

• To discover all the projects we have and who we need to contact
• To get lots of example data to use when formulating our data API

Plus, in the short run, we needed a full list for a separate purpose. I don't propose to maintain a centralized list. That's what we are moving away from. I just wanted to clear that up in case there was confusion.

The central list should be just a list of URLs (perhaps even self-discovered, though I'd be okay with explicit). This list might be important for ordering in the display. That list might be like:

http://arquillian.org/api/project.json

http://torquebox.org/api/project.json

http://jboss.org/jbossas/api/project.json

...

That could even be in a github repo somewhere.

[dan.j.allen]

Jason Porter wrote:

Honestly seems like some sort of script that should be run when we create a project (not that we do this all the time). The script would allow JIRA to be setup, add this information, maybe some other stuff as well. When everything was hosted like VCS then that would have been setup too. Maybe it's not such a good idea anymore. A simple datafile works just as well.

The initial project setup automation is good as an independent idea. That would save a lot of time getting a project setup. I recommend filing an issue for that. A few bits of input could then kick off a script and setup all the various infrastructure for a project, permissions, etc.

[dan.j.allen]

I think we can arm projects with some script raw material to generate the data file from sources like a pom.xml. But, I think it's reasonable to request for projects to maintain their data file. (If a project doesn't want to participate, then we'll just stick their file somewhere we can get at it until they want to maintain it).