Apache Derby 10.9.1.0 Release

Summary of Change

Truncation as part of implicit or explicit cast is now reported, either as exception or warning.

Symptoms Seen by Applications Affected by Change

In previous releases, implicit and explict casts from a character or binary string data type to a shorter data type would silently truncate the value if the receiving data type was not wide enough to hold the original value.

SQL now does correct checking of the length of character and binary strings, and will throw an exception in these two cases:

• Arguments to stored procedures and functions
• Values assigned to generated columns

Previously, if the actual value was longer than the datatype of the argument or column to which it was assigned, Derby would silently truncate the value and ignore the truncation. The SQL standard requires a truncation exception be thrown.

Derby now throws an SQLException with SQL state 22001 in these cases:

• Stored method calls that used to work, now throw SQLException with SQL state 22001.
• INSERT or UPDATE of tables with generated columns that used to work, now throw SQLException with SQL state 22001.

For other cases of explicit or implicit casts of string data types where the value is truncated, Derby now generates a DataTruncation warning (sub-class of SQLWarning) with SQL state 01004. The warning can be retrieved by calling getWarnings() on the ResultSet instance for SQL statements that return a ResultSet, or on the Statement instance for SQL statements that do not return a ResultSet.

Incompatibilities with Previous Release

Applications that use too long string values as arguments for stored methods, or use tables with generated columns for which the generated value is longer than the type of the generated column, may now see exceptions.

Applications that use implicit or explicit casts that truncate the value to make it fit in the receiving data type, may see warnings reported on the ResultSet or Statement.

Rationale for Change

The previous behavior could lead to inadvertent data loss, since the truncation was ignored. The previous behavior was not as per the SQL standard.

Application Changes Required

Applications should ensure that string values used as arguments to stored methods are within the range of the argument type.

Applications should ensure that string values assigned to generated columns are within the range of the argument type.

Example:

CREATE TABLE t_5749 (c VARCHAR(5) GENERATED ALWAYS
       AS ('--' || b), b VARCHAR(5))
INSERT INTO t_5749 VALUES (default, '12345')

The above code would work before, but silently truncate the generated value from '--12345' to '--123'. It will now throw SQLException with SQL state 22001.

If the old behavior is desired, the generation expression could be wrapped in a call to the CAST function:

CREATE TABLE t_5749 (c VARCHAR(5) GENERATED ALWAYS
       AS (CAST('--' || b AS VARCHAR(5))), b VARCHAR(5))
INSERT INTO t_5749 VALUES (default, '12345')

In that case, the INSERT statement would pass with no exceptions. Instead, the Statement instance on which it was executed will hold a DataTruncation warning with SQL state 01004, which can be retrieved by calling getWarnings() on the Statement.

Example:

s.executeUpdate("create procedure p5749 (a varchar(5)) modifies sql data 
    external name '...' language java parameter style java")
CallableStatement cs = conn.prepareCall("call p5749(?)");
cs.setString(1, "123456");
cs.execute();

The above code would work before, but silently truncate the generated value from '123456' to '12345'. The execute will now throw SQLException with SQL state 22001.

As with the previous example, it is possible to wrap the parameter in an explicit cast and get a warning on the CallableStatement instead of an exception:

CallableStatement cs = conn.prepareCall("call p5749(cast(? as varchar(5)))");
cs.setString(1, "123456");
cs.execute();

Summary of Change

Derby Network Server has been changed to reject client connections from clients that are not Derby Network Client (Those that do not send PRDID starting with DNC).

Symptoms Seen by Applications Affected by Change

Upon attempting to connect to Derby from a client other than Derby Network Client, users will get an immediate error. On the server side they should see an Invalid client and protocol message in the derby.log, e.g:

On the client side users will receive a protocol error related to the SYNTAXRM required value not found error sent by the server. The error will vary by connecting product. e.g. for the DB2 Universal JDBC Driver (JCC) this might be:

Incompatibilities with Previous Release

Clients that are not Derby Network Client, such as the DB2 Runtime Client and DB2 Universal JDBC driver will get a protocol error on connection on the client side and and a message that they are an Invalid Client in the derby.log. For prior releases the initial connection may have succeded giving the false impression that these were supported and would work in all cases.

Rationale for Change

There are currently no known supported network clients for Derby Network Server other than Derby Network Client. IBM support for the DB2 clients was discontinued after Derby 10.1. This change will avoid user confusion about support by ensuring that connections from non-supported clients to newer versions of Derby Network Server fail immediately and give an indication of the reason in the derby.log.

Application Changes Required

Applications should be modified to use Derby Network Client or if a non-Derby client is being developed, developers should submit a patch for Derby Network Server to accept the appropriate PRDID.

Summary of Change

The SQL state of the error returned from a lock timeout when the property "derby.locks.deadlockTrace=true" is set will return 40XL1 rather than 40XL2. 40XL1 is the documented SQL state for lock timeouts returned from Derby.

Symptoms Seen by Applications Affected by Change

In the previous release, applications which set "derby.locks.deadlockTrace=true" would get error 40XL2 when a lock timeout was received. This was confusing as simply setting the system to return extra diagnostic information would change the SQL state of the error returned by the system for a lock timeout.

Incompatibilities with Previous Release

Applications which explicitly tested for SQL state error 40XL2 as a lock timeout will no longer get that error. 40XL1 will now always be returned for a lock timeout error whether lock diagnostics are enabled or not.

Rationale for Change

The previous behavior was not intended. Lock timeout should generate same SQL state whether diagnostics are enabled or not. Only the text of the error will change based on the requested diagnostic level.

Application Changes Required

If the application has been explicitly coded to take action when getting error 40XL2 and does not do the same action for 40XL1, then the application should be changed. This action is usually retry logic based on a lock timeout or deadlock.

Summary of Change

getBytes and getString raise an exception if another getter has been invoked on the LOB column.

Symptoms Seen by Applications Affected by Change

In the previous release, applications were able to invoke getBytes and getString after invoking another getter on the LOB column first. In some cases this caused the data to be retured to be incorrect. Derby now raises an SQLExecption with state XCL18 when the application invokes any valid getter more than once on a given LOB column.

Rationale for Change

The previous behavior was inconsistent, and caused incorrect data to be returned to the application in some cases.

Summary of Change

A typo has been corrected in a column name of the DatabaseMetaData.getColumns() ResultSet.

Symptoms Seen by Applications Affected by Change

In previous versions of Derby, SCOPE_CATLOG was the name of the 19th column of the ResultSet returned by DatabaseMetaData.getColumns(). Now that column is named SCOPE_CATALOG. In addition, the ResultSet now has an extra trailing column, called SCOPE_CATLOG. The extra column duplicates the information in SCOPE_CATALOG and was added to reduce the risk of backward incompatibilities.

Incompatibilities with Previous Release

Applications will break if they expect SCOPE_CATLOG to be the name of the 19th column of that ResultSet or if they require that the ResultSet only contain 24 columns. Applications will display different columns names and longer rows if they pass this metadata on to end users.

Applications will only see these changes if they run against a Derby 10.9 engine. The version level of the engine can be retrieved by calling DatabaseMetaData.getDatabaseMajorVersion() and DatabaseMetaData.getDatabaseMinorVersion(). Applications will not see these changes if they run the Derby 10.9 network client against an older version of the Derby engine.

Rationale for Change

JDBC 4.1 has corrected a typo in the javadoc for DatabaseMetaData.getColumns(). Derby has renamed a column in order to conform to the latest version of the JDBC spec.

Application Changes Required

Applications which inspect this metadata by column position should not be affected: the contents of column 19 have not changed. Applications which inspect this metadata by column name should also not be affected. Note that the column contents are always null because the column only has meaning for databases which support both catalogs and reference types--Derby supports neither of these features.

Applications will need to be re-coded if they require that column 19 be named SCOPE_CATLOG or if they require that the ResultSet only contain 24 columns. Applications may need to be re-coded if they passthrough the metadata to end users.

Summary of Change

When running on Java 7 (or higher): when creating directories and files, the Derby server now by default limits access to the operating system account which booted the server from the command line.

Symptoms Seen by Applications Affected by Change

Other operating system accounts won't be able to access directories and file created by Derby. This is true regardless of whether looser security defaults were in place via umask(1) settings on Linux/Unix and default file visibility settings on Windows NTFS when the server was started.

Incompatibilities with Previous Release

In previous releases, the Derby server created directories and files using OS level access defaults--such as those specified by umask(1) settings on Linux/Unix and default file visibility settings on Windows NTFS. Now file access is limited to the operating system account which booted the server. Other accounts cannot view these files unless this new behavior is overridden as described below.

Note that this new behavior only affects the network server when it is started from the command line with Java 7 or higher. The new behavior does not by default affect server instances instantiated inside application code via the Derby API. Also, the new behavior does not by default affect embedded databases.

Rationale for Change

This change was made in order to make server-managed databases more secure.

Application Changes Required

Users can disable this new behavior by setting the system property derby.storage.useDefaultFilePermissions to true. Again, this is not necessary of running with Java 6 or lower, with embedded derby, or when the server is started programmatically inside application code via the API.

Conversely, by setting this system property to false, users can enhance security proactively by enabling the new behavior for embedded databases and databases managed by servers which were started programmatically inside application code via the Derby API (Java 7 and higher). The reason the default differs for the different deployment topologies is basically upwards compatibility concerns.

For information on how to set Derby system properties, see the Derby property reference section in the Derby Reference Manual.

Summary of Change

Streams obtained from a result set column are always implicitly closed when obtaining the value of the next column.

Symptoms Seen by Applications Affected by Change

An IOException may be raised when accessing streams obtained from a result set, whereas no exception was seen earlier.

Rationale for Change

The previous behavior was inconsistent, and, for some of the getter methods, in violation of the JDBC standard.

Application Changes Required

Read all the data in the stream before fetching the next column value.

Summary of Change

DERBY-1482 improved the resources used while procecessing triggers but this will apply only to the triggers created in databases that are created in 10.9 or higher OR to all triggers created after a database has been hard upgraded to 10.9 or higher.

Symptoms Seen by Applications Affected by Change

During trigger execution the system will continue to read all the columns from the trigger table if the trigger has been created prior to a hard upgrade to 10.9. For all the other triggers (ie the ones created in databases that are created in 10.9 or higher OR all triggers created after a database has been hard upgraded to 10.9 or higher), during trigger execution the system will now only read the columns necessary for the triggering sql and the sql associated with the executing trigger.

Rationale for Change

This work was done to improve the resource utilization during trigger execution. During trigger execution, the system will now only read the columns necessary for the triggering sql and the sql associated with the executing trigger. All the other columns columns from the triggering table will not be read into memory. This improvement is especially noticeable when the triggering table has large BLOB and/or CLOB columns. This work will affect triggers created in databases that are created in 10.9 or higher. This work will also apply to all triggers created after a database has been hard upgraded to 10.9 or higher. This work will not affect any triggers in databases running in soft upgraded databases with versions lower than 10.9.

Summary of Change

Derby may throw an XAException with a different error code when clients call javax.transaction.xa.XAResource.forget(Xid).

Symptoms Seen by Applications Affected by Change.

In previous releases, when clients called javax.transaction.xa.XAResource.forget(Xid) with an active transaction id, Derby raised an XAException with the XAER_NOTA error code. According to the XA+ standard, this error code is valid only for a heuristically completed transaction branch. Derby doesn't support heuristically completed transaction branches. In this situation, the correct error code is XAER_PROTO.

Incompatibilities with Previous Release

In this release, calling javax.transaction.xa.XAResource.forget(Xid) with an active transaction id causes Derby to throw an XAException whose error code is XAER_PROTO.

Rationale for Change

Derby's behavior was changed in order to conform with the XA+ Standard.

Application Changes Required

Applications which look for the XAER_NOTA error code after calling javax.transaction.xa.XAResource.forget(Xid) with an active transaction id should be re-coded to look for the XAER_PROTO error code instead.

Summary of Change

This release introduces a new NATIVE authentication scheme which is easier to administer than BUILTIN authentication.

Symptoms Seen by Applications Affected by Change

Applications which rely on BUILTIN authentication should upgrade to Derby 10.9 and use NATIVE authentication instead.

Incompatibilities with Previous Release

NATIVE authentication does not introduce any incompatibilities. It is simply a new feature. The Derby Developer's Guide describes NATIVE authentication in the section titled "Working with user authentication". Newly created databases can use this authentication scheme as is.

Rationale for Change

The BUILTIN authentication mechanism was originally intended just for testing purposes. Some problems mar it:

• Maintenance - By default, only the database owner can change passwords. In addition, passwords never expire. Also, the database owner can't list the known users. Applications must solve these problems themselves.
• System-wide Credentials - You can't securely store the BUILTIN credentials needed to authenticate system-wide operations like database creation, database restoration, and engine shutdown. Those credentials must either be stated on the JVM command line (where they can be snooped by tools like the Unix ps command) or stored in a human-readable derby.properties file.
• Duplicate Credentials - BUILTIN authentication lets you store multiple credentials for the same account. Two different BUILTIN user names access the same schema provided that the user names differ only by case.

These defects do not mar NATIVE authentication. NATIVE authentication is more secure and easier to administer than BUILTIN authentication.

Application Changes Required

Follow these steps to enable NATIVE authentication in an old database:

• Upgrade - Hard-upgrade the database to 10.9.
• Set message digest - Make sure that the derby.authentication.builtin.algorithm property is set in the upgraded database. NATIVE users can't be created until this property is set to a valid message digest algorithm. For information on how to set this property, please see the "Derby properties" section of the Derby Reference Manual.
• Create users - Create new users via the syscs_util.syscs_create_user() procedure. For information on how to invoke this procedure, please see the "Built-in system procedures" section of the Derby Reference Manual. The very first credentials you create must be for the database owner. Creating credentials for the database owner will turn on NATIVE authentication.
• Bounce database - Shutdown the database and bring it back up. NATIVE authentication should be enabled.