State of incremental compilation of IOC bootstrapper

[jannis]

Hello,

we just updated to GWT 2.7 and got it running. We can experience the benefits of incremental builds as long as we are not touching any file that leads to the generation of the IOC bootstrapper. If we do so, the complete bootstrapper will be generated again which takes some time. Thus, we are wondering if the incremental generation of the bootstrapper should work by now and we are doing something wrong or if it is currently under development.

Regards

Jannis

[csa]

Hi Jannis,

In 3.0.2 and 3.1.0 we introduced new generator caches to cause regeneration only when necessary. In most cases the decision whether or not to regenerate is fairly simple, except for the IOC bootstrapper generator. So, currently there are cases where the bootstrapper is regenerated where it wouldn't have to be (i.e. when a remote interface changes). This is something we plan to improve in the short-term.

However, changing the bootstrapper generator so it generates individual files for different bean graph partitions and then consequently only regenerating the files affected by a change is a way bigger effort. We haven't solved this yet and at this point it's not clear whether or not we'll have the resources to implement this. Do you have an idea how to solve this? Would you be interested in helping out?

Cheers,

Christian

[jannis]

Hi Christian,

thanks for the fast reply. We are happy about the caching and it works fine.

Just to give you a figure of the impact of the generation of the IOC bootstrapper, compilation of a change in an IOC independent file takes 3 seconds whereas compilation with generation of bootstrapper takes 30 seconds.

The use case we currently have is to improve the UI development. We are making use of templated views and data fields. Our UI developers are always complaining about the long compilation times (compared to other frameworks like angular) and thus we would like to offer them the benefits of incremental compilation.

I have two ideas on this

- If there are changes in html files only, often no generation of the bootstrapper is required (AFAIK)

- If there are changes that have only local effects (changing of data field bindings), it would be nice to regenerate the affected methods of the bootstrapper only

So what I think of is to get the IOC relevant changes of changed files and evaluate the required action on bootstrapper generation (no, partial, complete).

Depending on the size of bean graph partitions the two approaches might complement each other.

Regards

Jannis

[csa]

Hi,

Changes to HTML files/templates do not cause a regeneration of the bootstrapper, only changes to @Templated classes will. Just making sure you're not experiencing some sort of bug.

I completely agree that a 30 seconds refresh is horrible which is why in the short-term we have to make sure to generate the bootstrapper only when absolutely necessary.

Based on my measurements it isn't the actual code generation but building up all the data structures that takes most of the time. If say a new type is introduced or deleted it could affect the bean graph in various ways. We don't really have a good way currently to see what needs to be redone given a certain change. Solving this might actually lead to a design where we generate smaller units instead of one monolithic bootstrapper. So, we haven't found a "quick" solution to partial regeneration that doesn't involve a significant redesign but that doesn't mean there isn't one. Ideas welcome!

Cheers,

Christian

[jannis]

Hi,

we are currently behind the latest version of Errai. So I will update our fork in the next days and check if that solves the problem of regeneration for HTML files only changes.

By now, I don´t have a good knowledge on which information is available during recompile and which is not. The first question for me would be whether it is possible to know what changes were made. As far as I understand it is possible to get a list of files that were changed. The next step for me would be to get to know what changes where made. According to the generation of the bootstrapper, those were adding / removing injections, adding / removing data field bindings and similar. Next, I don´t know if you can reuse the former bean graph and apply the changes, remove a link here, add one there and then do the (complete) code regeneration.

But those thoughts only relate to "simple" changes. Introducing a new type requires much more work.

Regards

Jannis

[jannis]

Hi Christian,

I was wondering whether there are any new insights or plans from your side concerning faster IOC bootstrapper generation?

We are still very happy to use errai and are relying heavily on it. It improves our UI development as we are able to save some extra work by directly using HTML templates. But the bootstrapper generation performance is a show stopper for us in the long run. Thus it is important for us to know if we can expect any improvements during the next half year. Otherwise we might have to consider to switch to another framework. Especially as the UI development productivity was one of the main topics of the GWT.create.

I tried to extract the relevant information from the recompilation log, which you find below. If there is anything else I can provide you to help, please let me know.

Regards

Jannis

[INFO]          Found 6804 cached/archived units.  Used 6804 / 6806 units from cache.

[INFO]          Invalid units found: 4

[INFO]          Compiling...

[INFO]          Computing all possible rebind results for 'org.jboss.errai.ioc.client.Bootstrapper'

[INFO]             Rebinding org.jboss.errai.ioc.client.Bootstrapper

[INFO]                Invoking generator org.jboss.errai.ioc.rebind.ioc.bootstrapper.IOCGenerator

[INFO]                   generating ioc bootstrapping code...

[INFO]                   generating IOC bootstrapping class...

[INFO]                   setting up injection context...

[INFO]                   Checking ErraiApp.properties for configured types ...

[INFO]                   injection context setup in 39ms

[INFO]                   generated IOC bootstrapping class in 9872ms (3633 beans processed)

[INFO]          Adding '1' new generated units

[INFO]             Compiling...

[INFO]                Compilation completed in 12,66 seconds

[INFO]             Removing invalidated units

[INFO]             Resolving org.jboss.errai.ioc.client.BootstrapperImpl.[....]

[INFO]          Unification traversed 49804 fields and methods and 8051 types. 4779 are considered part of the current module and 4779 had all of their fields and methods traversed.

[INFO]          [WARN] Some stale types ([org.jboss.errai.ioc.client.BootstrapperImpl$47$1$13$1$3, [...]]) were not reprocessed as was expected. This is either a compiler bug or a Generator has legitimately stopped creating these types.

[INFO]          Compiling 1 permutation

[INFO]             Creating PermutationWorkerFactory instances

[INFO]             Compiling permutation 0...

[INFO]             Linking per-type JS with 4779 new types.

[INFO]                prelink JS size = 10416131

[INFO]                prelink sourcemap = 10416131 bytes and 169206 lines

[INFO]                postlink JS size = 34006449

[INFO]                postlink sourcemap = 34006449 bytes and 635362 lines

[INFO]             Creating split point map file for the compile report

[INFO]             Source Maps Enabled

[INFO]             Permutation took 2635 ms

[INFO]          Compile of permutations succeeded

[INFO]          Compilation succeeded -- 34,315s

[csa]

Hi Jannis,

The latest 3.2.0 and 3.1.2-SNAPSHOTs actually contain optimizations that made a difference for the apps that we tested. Can you try those and see how it affects your initial boostrapper generation but more importantly your refreshes?

We have more optimizations lined up to speed up refreshes for 3.1.2.Final, but I think reimplementing IOC for incremental generation is off the table for at least the next 3 months.

Cheers,

Christian