Simplicty and flexibility!

DataEase 8.5 - BETA 1 - Preview

When we do big structural changes to the product, we prefer to run it through a short Beta program before we release the improvements.

This is still an early Beta of 8.5 but as there is some really significant updates/changes we would like to get it out there as soon as possible to get feedback on the improvements.

Download 8.5 BETA 1

First of all. Sorry to all of you that though that the next version would be 8.3!

It NEEDED to be 8.5 when we realized the difference we have made, and to illustrate the difference we gave it the code-name BlackBird after the SR-42 since in comparison if DE8.2 is a horse, then DE8.5 is a sportscar but with the afterburner it is a Blackbird (Mach 3).

When we realised that Blackbird in Russian is Drozd the name stuck as our head of DE8 development is Andrey Drozdov, and he is the one to thank for this fantastic improvement.

Up to and including 8.2 we have focused on functional improvement. With 8.3 we decided that we needed to take a look at performance. Before 8.0 we weren't able to us DFW for anything so performance was not a real problem ... but with 8.0 the fun was back in DE development so even things went "blindingly" with ExecDQL etc, there was obviously something wrong. Some things just STOPPED!

It was clear that something was "snagging" deep down in prism and that the problem had geometric development. Small to medium sized transaction went "OK" but suddenly things just stopped turning over. Another serious problem was that it seems that repeated jobs went slower and slower.

When we drew a line under 8.2 and started on 8.3 we decided ti was time to get dirty so we jumped in on the deep end and did we get pay dirt.

No reason to dwell on past mistake but it was a mess as you all suspected.

We cleaned up the locking model and started on a crazy index updating regime. IT WAS a hard job, because so many people have tried to fix it over the years without really knowing what they were doing so under layers of unnecessary, crazy and straight out dangerous pieces of code we finally managed to get to the core.

It didn't help that someone in charge of development was "hellbent" on Locking and wanted to make sure that nothing was changed or updated without permission in triplicate.

Our belief is that things should just work. No error message, no explanations or pop-ups - just work! So if you want to update a table or an index when someone else is looking at it, so you should.

To make a long story short:

DE 8.5 is dramatically much faster than any previous version of DFW including the cousin Ffenics.

Today we ran a test on the following DQL:

define "startTime" time .

define "retval" text .

define "counter" number.

counter := 0 .

startTime := current time .

For SomeData ;

modify records in SomeData named "Drozd" with RecordNr = SomeData RecordNr

SomeText := concat(counter," This is changed!" ).

counter := counter + 1 .

end

message concat("Modify took : ", current time-starttime, " seconds.") window .

On a table with 100.000 records (Shared database).

The results was as follows:
DE8.2

Product Seconds to complete Faster than 8.2 Faster than Ffenics 1.63
DataEase 8.2 1441 1 0.3
Ffenics 1.63 396 3.64 1
DataEase 8.5 - Full Mode 118 12.2 3.36
DataEase 8.5 - Manual Indexupdate (index off/manual update) 13 111 30.46

Yes, you are reading it right.

DataEase 8.5 is in "normal" mode 12 times faster than 18.2 for this update and 111 times faster if you use the "afterburner".

Truly a Blackbird!

This is not a finished beta, but please download and test it. It will still retain elements of 8.3 and 8.2 but the core functionality is all 8.5

The main focus has been on DQL inserting/updating and deleting but due to radical changes in locking and indexing engines it is also faster EVERYWHERE!!! from searching to looking up data to displaying data in subforms etc. etc.

Have a go and give us your feedback!


1. Tab Navigation added to Main View.

In 8.5 we have changed the default opening of forms to Maximized and added tab navigation to move between pages. The old Windows model was a mess, and we are now moving more in line with contemporary thinking. where one see one page on screen at the time and make it easy to move between them.

You can still open forms normalized and use this model. This is an early BETA so how this work will change/improve. There is some bugs like the wrong tab being active etc, we are aware of this so don't report this.


2. Dramatic speed improvement of bulk Insert/Update/Delete

Download Sample

PS! Remember that this sample needs to be opened from 8.5 Beta, Startup.exe will not open 8.5 Beta, but 8.2 Full.

The main change is to bulk Insert/Update and Delete.

It is no secret that for a long time DFW have had serious performance problems when running modify/delete on a large set of data. This problem as also had a logarithmic evolution i.e. the same job will take longer and longer when you repeat it on the same data set.

One part of the problem is basically how DataEase has solved challenges with locking and updates of data and indexes, which to be honest hasn't been great.

As long as we run DataEase as a file based database there will be challenges with these things as many different processes open the same files compared to a Server based approach where the Server do all the file management and only server the data.

The main performance problem is down to index updates, which in the old model is updated for each insert, modify, delete. In the new model we have added the function IndexUpdate() which will allow you to control index updating in your app more specifically.

We are still working with the most optimum model, but we would like you to try it out to see if you can get performance improvements.

We have got improvements from 2000% to 5000% so far, but this is preliminary (20 to 50 times improvement in speed) where jobs that would take 20 or 50 seconds in old model will now take 1 sec (for both). As 1 sec is the lowest clock-able time in DE we don't have the exact answer...

At the moment IndexUpdate() have the arguments: "On","Off","TableName", this is likely to be changed as we also want to include a setting for old modle ("Single") and one for "Quick" i.e. the current setting of Off.

At the moment "Off" means that the index is Bulk updated at the end of the transaction, while "Off" in the future will mean OFF, Quick will then do the Bulk update at the end of the transaction.

IndexUpdate("Off") is global (do not apply to form mode) so you need to switch it back on again at the end of your DQL if you don't want it to apply to future DQL executions in same session.

We have included a simple app to show you the performance improvement. It is very rudimentary as it is the app we test it on in-house ;-)

1. Here you simple delete the entire data-set. There is some problems with table locking so at the moment it will not work if you have more than one User using the same table, or after you have run the Delete/Modify. We are looking into this problem now. This is simply tor reset the app.

2. Hit populate. We use 100.000 but you can use more or less. Now you will have a data-set with 333.333 Andrey, Ulrik, Benny.

3. This is just to preview what is the data in the table. If you try to run this sample in a PRe 8.5 version of DataEase the loading here will take forever. In 8.5 it is quick as...

4. Here you delete data with a For/delete records ( in same table as the for i.e. no with on the Delete Records). Try to delete with and without IndexUpdate On/Off.

5. Same as 4 but now we do modify records with i.e. without for loop. Repeat test.

6. Here we Modify Records with For/Modify Records.

7. Here we do it directly with Modify Records in with.

You will see dramatic improvement in speed. Try it in Network and you will be shocked ;-)

Known problems:

There is still problems with other sessions recognizing/reloading index after bulk update on another computer. The response will be that the other sessions will search sequential. This only apply if the table was open when it was modify in another session.

Problem with locked table after update so you can't do delete records in tablename. (without with).


3. Orientation on Button added.

Buttons are very useful as they have actions as well as OML events. One problem has been that if you want to use them for Menus etc, they are always Center Aligned.

To fix this has been on our list for a long time, and finally we got around to do it.

You can now align them the same way you can any other text.

Left, Center, Right.


Full Change list so far in 8.5
r1744

Update to Livery. Worng deres.dll and STARTUP.TIF

---------------------

r1743

Livery Update 8.5 beta. deresdlg.dll deres.dll startup.tiff dataease.ico

---------------------

r1742

One more test commit to test the way to update indexes in multiuser environment; index update routine is fully changed . NB! NOT FOR CUSTOMER'S BUILD!

---------------------

r1741

One more test commit to test the way to update indexes in multiuser environment; instant saving is removed. NB! NOT FOR CUSTOMER'S BUILD!

---------------------

r1740

One more test commit to test the way to update indexes in multiuser environment. NB! NOT FOR CUSTOMER'S BUILD!

---------------------

r1739

Change request 1400 (new prism function list) is improved, the problem with recognition of opkey as Numeri cstring is fixed

---------------------

r1738

Test commit to test the way to update indexes in multiuser environment. NB! NOT FOR CUSTOMER'S BUILD!

---------------------

r1737

CR 1483 (speeding up of bulk operations) is improved,index update locking during bulk operations is removed. Also some outdated comments are removed.

---------------------

r1736

Remove the restrictions on maximize minimize windows that was added to tabbed windows.

---------------------

r1735

Update version number to 8.5

---------------------

r1734

CR 1483 (speeding up of bulk operations) is improved, the problem with IndexUpdate(FormName) is fixed

---------------------

r1733

CR 1483 (speeding up of bulk operations) is improved, keys On is added to duplicate Full, also key None is replaced by Off

---------------------

r1732

CR 1483 (speeding up of bulk operations) is improved, keys Full/Quick are added to IndexUpdate insteade of On/Off, also key None is implemented

---------------------

r1731

Fix install to work with new font.

---------------------

r1730

Update av Installation, readme etc.

---------------------

r1729

CR 1483 (speeding up of bulk operations) is changed, IndexUpdate (TableName) functionality is implemented

---------------------

r1728

And the rest of the version files to 8.3 as well.

---------------------

r1727

Updated version to 8.3

---------------------

r1726

Update to 8.3 beta

---------------------

r1725

CR 1483 (speeding up of bulk operations) is changed, IndexUpdate (On/Off) functionality is changed to switch On/Off single update; debug message is switched off. NB! IndexUpdate(TableName) does not work yet!

---------------------

r1724

CR 1483 (speeding up of bulk operations) is improved, IndexUpdate (On/Off/TableName) is implemented for On/Off.

---------------------

r1723

CR 1483 (speeding up of bulk operations) is improved, skeleteon for IndexUpdate (On/Off/TableName) is added. NB! Intermediate build - no internal functional pf IndexUpdate yet!

---------------------

r1722

Change request 1400 (new prism function list) is improved, SetState is changed to support key words show/hide/enable/disable/click

---------------------

r1721

One more test commit to test the way to improve performance. Single IDX update is temporary restored NB! NOT FOR CUSTOMER'S BUILD! NEW LOCKING WAY SHOULD BE TESTED

---------------------

r1720

One more test commit to test the way to improve performance. Cosmetic changes, misprinting in error message is corrected NB! NOT FOR CUSTOMER'S BUILD! NEW LOCKING WAY SHOULD BE TESTED

---------------------

r1719

One more test commit to test the way to improve performance. Debug messages for nulled IDX field ID is added, bulk IDX update is changed to single for record adding NB! NOT FOR CUSTOMER'S BUILD! NEW LOCKING WAY SHOULD BE TESTED

---------------------

r1718

CR 1483 (speeding up of bulk operations) is prelimenary fixed; changes tested in test commits re implemented

---------------------

r1717

One more test commit to test the way to improve performance. Debug messages are removed NB! NOT FOR CUSTOMER'S BUILD! NEW LOCKING WAY SHOULD BE TESTED

---------------------

r1716

One more test commit to test the way to improve performance. Problems with bulk modification is fixed NB! NOT FOR CUSTOMER'S BUILD! REMOVE DEBUG MESSAGE BOXES!

---------------------

r1715

One more test commit to test the way to improve performance. Bulk indexes update is added, debug message are modified to tell the name of form NB! NOT FOR CUSTOMER'S BUILD! REMOVE DEBUG MESSAGE BOXES!

---------------------

r1714

One more test commit to test the way to improve performance. Indexes update optimized for modify records, one more debug message box for is added NB! NOT FOR CUSTOMER'S BUILD! REMOVE DEBUG MESSAGE BOXES!

---------------------

r1713

One more test commit to test the way to improve performance. Debug message box control is improved NB! NOT FOR CUSTOMER'S BUILD! REMOVE DEBUG MESSAGE BOXES!

---------------------

r1712

One more test commit to test the way to improve performance. Indexes update optimized for add records, debug message box is added NB! NOT FOR CUSTOMER'S BUILD! REMOVE DEBUG MESSAGE BOXES!

---------------------

r1711

One more test commit to test the way to improve performance. NB! NOT FOR CUSTOMER'S BUILD!

---------------------

r1710

One more test commit to test the way to improve performance. NB! NOT FOR CUSTOMER'S BUILD!

---------------------

r1709

Test commit to test the way to improve performance. NB! NOT FOR CUSTOMER'S BUILD!

---------------------

r1708

CR 1479 (Change Request - add Button UI properties) is implemented

---------------------

r1707

Change request 1376 (new functions StringReplace and SetValue) is improved, the problem wit SetValue error after for, modify and delete is fixed. Also some cosmetic changes to improve code reading

---------------------

r1706

Remove the open as dialog when form is opened from inside another form and window not maximized.

---------------------

r1705

Tabbed window default.

---------------------

r1704

Change request 1474 (Delete tmp files) is improved, temporary aweb files are are removing after 48 hours of last access; some definition are moved to common headers

---------------------

r1703

Cosmetic commit - outdated comments are removed

---------------------

r1702

Change request 1474 (Delete tmp files) is improved, temporary aweb files are moved to app\temp directory to be ready for easy removing

---------------------

r1701

Bug 1482 (GPF after *ExecDQL functions) is fixed

---------------------

Published: 31/01/15 - 13:07:00 (Amanda Higgins)

Related Articles

DataEase 8.5 - What is new! (Page 1) - Roughly 8.1

This is the first page of 3 where we recapitulate the Blog entries made by our development team through the development from pre 8.0 to 8.5. We have edited and updated the entries, but the appear in the order they were made and with the ideas, experienc...

DataEase 8.5 - What is new! (Page 2) - Roughly 8.2

This is the Second page of 3 where we recapitulate the Blog entries made by our development team through the development from pre 8.0 to 8.5. We have edited and updated the entries, but the appear in the order they were made and with the ideas, experien...

DataEase 8.5 - What is new! (Page 3) - Roughly 8.5

This is the third page of 3 where we recapitulate the Blog entries made by our development team through the development from pre 8.0 to 8.5. We have edited and updated the entries, but the appear in the order they were made and with the ideas, experienc...

NEW! ?Functions in DataEase 8.0 to 8.5 (Page 2)

Some of you might be aware of it but it seems a lot isn't so it might be time to advertise a little the vast number of new functions and functionalities that has been added to DataEase 8.0 through 8.5. For a long time the only way new functions found the...

NEW! Functions in DataEase 8.0 to 8.5 (Page 1)

Some of you might be aware of it but it seems a lot isn't so it might be time to advertise a little the vast number of new functions and functionalities that has been added to DataEase 8.0 through 8.5. For a long time the only way new functions found the...

DataEase 8.5 - BETA 1 - Preview