How can I create a new control that is derived from CGXMaskControl? I want to extend the mask character set  and modify the behavior.

Here are some steps how you can subclass CGXMaskControl:

Derive class from CGXMaskData. Make sure to insert DECLARE_DYNCREATE in the class declaration.

Override the IsCharValid and IsMaskChar member functions.

When registering the mask control in OnInitialUpdate, pass runtime class information for your CGXMaskData derivative to the CGXMaskControl constructor.

Example:

RegisterControl(
      GX_IDS_CTRL_MASKEDIT,
      new CGXMaskControl(this, GX_IDS_CTRL_MASKEDIT, RUNTIME_CLASS(CMyMaskData)));

The IsCharValid is the method that let's you validate and adjust individual characters.