HDF version 4.4.0 currently under development
====================================================

INTRODUCTION

This document describes the differences between this release and the
HDF 4.3.1 release.  It is written for people who are familiar with
previous releases of HDF and wish to migrate to this version of HDF.

Note that the HDF4 documentation will be updated at the time of
each final release and can be found on the HDF4 support page at:

    https://portal.hdfgroup.org/display/HDF4

The official HDF4 releases can be obtained from:

    https://portal.hdfgroup.org/downloads/index.html

If you have any questions or comments, please send them to the HDF Help Desk:

     help@hdfgroup.org

CONTENTS

- Breaking changes
- New features and changes
  -- Configuration
  -- C Library
- Bugs fixed since HDF 4.3.1
  -- Configuration
  -- Library
  -- Utilities
- Documentation
- Platforms Tested
- Known problems


Breaking changes
================

    - Vgroup API prototype changed: Vgetname, Vgetclass, Vinquire

      The unbounded write in Vgetname, Vgetclass, and Vinquire could
      previously overflow caller-supplied buffers.  The signature of these
      functions has been changed to take a size_t *buf_size (IN/OUT) parameter
      to address that security vulnerability. In addition, Vgetnamelen
      and Vgetclassnamelen were removed because applications can now use
      Vgetname/Vgetclass to query the required length instead.

      The new prototypes are:
        int Vgetname(int32 vkey, char *vgname, size_t *buf_size);
        int Vgetclass(int32 vkey, char *vgclass, size_t *buf_size);
        int Vinquire(int32 vkey, int32 *nentries, char *vgname, size_t *buf_size);

      Callers must update all call sites accordingly by calling
      Vgetname/Vgetclass the first time passing in NULL for the buffer
      to retrieve the size of the name/class.  The size then must be
      incremented to account for the null-terminator prior to being
      passed into the fetch call.

      The Fortran bindings (VFGNAM, VFGCLS, VFINQ) had no buffer-length
      information anywhere in the call chain and were independently
      exposed to the same overflow. This was fixed as part of the
      same change — the Fortran stubs now query the actual length
      internally and allocate correctly-sized buffers before copying
      into the caller's CHARACTER variable.

      For the Java API, no changes to public method signatures —
      buf_size handling is fully internal to the JNI layer. The
      old JNI implementations exposed Java applications to the same
      overflow risk as C by writing directly into fixed-size buffers
      (H4_MAX_NC_CLASS/H4_MAX_GR_NAME). The JNI layer now queries the
      actual required length and allocates correctly-sized buffers
      before fetching.

      Fixes GitHub issue [#872](https://github.com/HDFGroup/hdf4/issues/872)


New features and changes
========================
    Configuration:
    -------------
    - Removed support for autotools build system.

    - The NetCDF interface has been removed from both C and Fortran

      The HDF4 multi-file interface is built on NetCDF 2.3.2, which was
      copied and modified instead of used as a library. HDF4 then exported
      this API for public use.

      This exported NetCDF API causes conflicts when both HDF4 and NetCDF
      are used in the same application, so name-mangling hacks were added to
      hide the NetCDF symbols. Since most people now enable the name mangling
      scheme (via --disable-netcdf, etc.) and get their NetCDF API from
      Unidata, the HDF4 NetCDF 2.3.2 API has been removed from both the C
      and Fortran exported API.

      Autotools:
        * The --enable-netcdf option has been removed
      
      CMake:
        * The HDF4_ENABLE_NETCDF option has been removed

      C API:
        * The hdf2netcdf.h and hdf4_netcdf.h header files are no longer
          distributed with the library

        * The following variables have been removed:
            - ncerr
            - ncopts

        * The following functions have been removed:
            - nccreate
            - ncopen
            - ncredef
            - ncendef
            - ncclose
            - ncinquire
            - ncsync
            - ncabort
            - ncdimdef
            - ncdimid
            - ncdiminq
            - ncdimrename
            - ncvardef
            - ncvarid
            - ncvarinq
            - ncvarput1
            - ncvarget1
            - ncvarput
            - ncvarget
            - ncvarputs
            - ncvargets
            - ncvarputg
            - ncvargetg
            - ncvarrename
            - ncattput
            - ncattinq
            - ncattget
            - ncattcopy
            - ncattname
            - ncattrename
            - ncattdel
            - nctypelen
            - ncsetfill
            - ncrecinq
            - ncrecget
            - ncrecput
            - ncnobuf

        * The following typedefs have been removed:
            - nc_type
            - nclong

        * The following #defined symbols have been removed:
            - NC_RDWR
            - NC_CREAT
            - NC_EXCL
            - NC_INDEF
            - NC_NSYNC
            - NC_HSYNC
            - NC_NDIRTY
            - NC_HDIRTY
            - NC_NOFILL
            - NC_LINK
            - NC_FILL
            - NC_NOWRITE
            - NC_WRITE
            - NC_CLOBBER
            - NC_NOCLOBBER
            - NC_UNLIMITED
            - NC_GLOBAL
            - NC_UNSPECIFIED
            - NC_BYTE
            - NC_CHAR
            - NC_SHORT
            - NC_LONG
            - NC_FLOAT
            - NC_DOUBLE
            - NC_BITFIELD
            - NC_STRING
            - NC_IARRAY
            - NC_DIMENSION
            - NC_VARIABLE
            - NC_ATTRIBUTE
            - NC_NOERR
            - NC_EBADID
            - NC_ENFILE
            - NC_EEXIST
            - NC_EINVAL
            - NC_EPERM
            - NC_ENOTINDEFINE
            - NC_EINDEFINE
            - NC_EINVALCOORDS
            - NC_EMAXDIMS
            - NC_ENAMEINUSE
            - NC_ENOTATT
            - NC_EMAXATTS
            - NC_EBADTYPE
            - NC_EBADDIM
            - NC_EUNLIMPOS
            - NC_EMAXVARS
            - NC_ENOTVAR
            - NC_EGLOBAL
            - NC_ENOTNC
            - NC_ESTS
            - NC_EMAXNAME
            - NC_ENTOOL
            - NC_EUNLIMIT
            - NC_EXDR
            - NC_SYSERR
            - NC_FATAL
            - NC_VERBOSE

        * The following #defined symbols were moved to mfhdf.h from hdf4_netcdf.h:
            - FILL_BYTE
            - FILL_CHAR
            - FILL_SHORT
            - FILL_LONG
            - FILL_FLOAT
            - FILL_DOUBLE

      Fortran API:
        * The netcdf.inc and netcdf.f90 files are no longer distributed with the library

        * The following functions have been removed:
            - nccre
            - ncopn
            - ncddef
            - ncdid
            - ncvdef
            - ncvid
            - nctlen
            - ncsfil

        * The following variables have been removed:
            - NCRDWR
            - NCCREAT
            - NCEXCL
            - NCINDEF
            - NCNSYNC
            - NCHSYNC
            - NCNDIRTY
            - NCHDIRTY
            - NCLINK
            - NCNOWRIT
            - NCWRITE
            - NCCLOB
            - NCNOCLOB
            - NCGLOBAL
            - NCFILL
            - NCNOFILL
            - MAXNCOP
            - MAXNCDIM
            - MAXNCATT
            - MAXNCVAR
            - MAXNCNAM
            - MAXVDIMS
            - NCNOERR
            - NCEBADID
            - NCENFILE
            - NCEEXIST
            - NCEINVAL
            - NCEPERM
            - NCENOTIN
            - NCEINDEF
            - NCECOORD
            - NCEMAXDS
            - NCENAME
            - NCENOATT
            - NCEMAXAT
            - NCEBADTY
            - NCEBADD
            - NCESTS
            - NCEUNLIM
            - NCEMAXVS
            - NCENOTVR
            - NCEGLOB
            - NCENOTNC
            - NCFOOBAR
            - NCSYSERR
            - NCFATAL
            - NCVERBOS
            - NCENTOOL

    - Corrected usage of FetchContent in the HDFLibMacros.cmake file.

      CMake version 3.30 changed the behavior of the FetchContent module to deprecate
      the use of FetchContent_Populate() in favor of FetchContent_MakeAvailable(). Therefore,
      the copying of HDF specialized CMakeLists.txt files to the dependent project's source
      was implemented in the FetchContent_Declare() call.

    - Renamed HDF4_ENABLE_USING_MEMCHECKER to HDF4_USING_ANALYSIS_TOOL

      The HDF4_USING_ANALYSIS_TOOL is used to indicate to test macros that
      an analysis tool is being used and that the tests should not use
      the runTest.cmake macros and it's variations. The analysis tools,
      like valgrind, test the macro code instead of the program under test.

    - The HDF4-build ncgen and ncdump tools have been removed

      We no longer provide these tools through HDF4. They were wildly
      out-of-date (circa 1993) and we find no value in maintaining ancient
      versions of tools that are better obtained elsewhere.

      If you need ncdump or ncgen, you should get them from a netCDF
      distribution. The official version of these tools can be found
      on GitHub or via your favorite package manager.

        https://github.com/Unidata/netcdf-c
 
    C Library:
    ----------
    - Removed intn and uintn from headers

      These types represent native integers and unsigned integers and have
      been typedef'd to int and unsigned in for a long time. They are
      retained in hdf.h so old code will compile, but other public headers
      now use int and unsigned in place of these types.


Bugs fixed since HDF 4.3.1
===========================

    - Removed memory issues in low-level Vgroup code

      The code that destroys the atom groups for Vdatas and Vgroups was
      placed after the free lists had already been freed, so, that code
      added more released memory to the free lists, which were never
      freed, causing many memory leaks.  This is now fixed.

      Fixes GitHub issue [#819](https://github.com/HDFGroup/hdf4/issues/819)

    - Removed an invalid read in the DD management code

      HTIcount_dd had a block of incorrect code that caused invalid read
      errors. This is now fixed.

      Fixes GitHub issue [#808](https://github.com/HDFGroup/hdf4/issues/808)

    - Fixed SDgetcompinfo incorrect behavior

      When SZIP library is not available, SDgetcompinfo() fails even
      though it only needs to retrieve compression metadata, not actually
      compress/decompress data. The function should succeed in returning
      compression information regardless of whether the SZIP library
      is present.

      This restriction is now removed from HCIinit_coder() to allow
      metadata queries without the szip library.

      Fixes GitHub issue [#867](https://github.com/HDFGroup/hdf4/issues/867)

    - Fixed a bug introduced by a previous fix in 3.1 release

      When opened files were closed in different order than when they
      were created, segfault and memory leaks occurred.  This fix
      checks _curr_opened == 0 and resets _ncdf and _curr_opened to 0
      in ncreset_cdflist on teardown, which would have been correct in
      the previous fix.  The fix has been verified with an out-of-order
      LIFO file-closure program under Valgrind.

      Fixes GitHub issue [#862](https://github.com/HDFGroup/hdf4/issues/862)

    - SDreaddata fails with DFACC_RDWR/no encoder combo

      Reading szip-compressed data from files opened with DFACC_RDWR
      when only decoder is available may fail. This is a pre-existing
      architectural issue where file open mode is conflated with operation
      type. This issue is now fixed by updating the internal function
      hdf_get_vp_aid() to take the correct access mode and all the call
      sites to provide the appropriate one.

      Fixes GitHub issue [#870](https://github.com/HDFGroup/hdf4/issues/870)

    - Fixed memory issues in hdiff and hrepack

      There were a large number of unchecked API calls and unreleased
      resources in hdiff and hrepack. They are now fixed.

    - Removed a potential overflow in DAset_elem

      Changed to use size_t instead of int to avoid overflowing a 32-bit
      int in DAset_elem.

    - Removed a potential double free situation

      Reset a pointer NULL after free in vimakecompat() to avoid a possible
      double free situation.

    - Fixed an incorrect replacement of a memset

      The memset was placed in done: in HAIget_atom_node(), causing the
      memset to happen in the case of failures. This is now fixed.

    - Fixed an integer overflow and a memory leak

      They occurred in the DD management code. This is now fixed. A
      utility function, HTPmemory_cleanup() was also added for future
      uses.


Documentation
=============


Platforms Tested
================
This version has been tested in the following platforms:


    Linux 6.17.0 x86_64              AMD AOCC 5.2.0 (clang/flang)
    Ubuntu 24.04                     GNU gcc 13.3.0
                                     gcc (distro default) ppc64le via QEMU
                                     MinGW-w64 cross, GNU GCC 13.x
                                     NVIDIA HPC SDK / NVHPC 26.5.0 (CUDA 13.2)

    Linux 6.8.0 x86_64               clang 18.1.8
    Ubuntu 22.04                     Address/Leak/UB sanitizers

    macOS 26 (Tahoe) arm64           Apple clang 21.0.0

    Windows Server 2025              ClangCL 18.1.8
                                     Cygwin GNU gcc 14.4.0
                                     MSVC 19.51, Visual Studio 18 2026, x64
                                     MSYS2 mingw32 / mingw64 / 
                                         ucrt64 (GNU GCC 16.2.0) / 
                                         clang64 (Clang 22.1.8)

    Windows Server 2022              MSVC, Visual Studio 17 2022, x64
                                     Intel oneAPI 2025.2 (icx)


Known problems
==============
o  Valgrind may report "Conditional jump or move depends on uninitialised
   value(s)" in JPEG-related test comparisons when the library is built
   against SIMD-enabled libjpeg-turbo. This is a documented false positive
   caused by Valgrind mis-tracking libjpeg-turbo's SSE2 color-space
   conversion code, not an actual bug in HDF4. It does not occur when
   built/run with SIMD disabled (JSIMD_FORCENONE=1). See the libjpeg-turbo
   README, "SIMD Extensions" section at:
    https://github.com/libjpeg-turbo/libjpeg-turbo/blob/main/README.md#memory-debugger-pitfalls
   for details.

o  The test_sd() test in mfhdf's hdftest can fail depending on the privileges
   of the account running the test. We've engineered around cases like running
   as root, but you may still encounter test failures when running that test
   from highly privileged accounts that can open read-only files for writing.

o  The Fortran interface does not work on 64-bit systems as it stores addresses
   in memory as Fortran INTEGER values, which are typically 32-bit. The
   Fortran interface is currently disabled by default due to this. It should
   only be enabled on 32-bit systems.

o  CMake files do not behave correctly with paths containing spaces.
   Do not use spaces in paths because the required escaping for handling spaces
   results in very complex and fragile build files.
   ADB - 2019/05/07

o  Several Fortran examples print "^@" when displaying strings (for example,
   names of the attributes). This happens because Fortran application
   doesn't know the length of the strings passed from the C library.
   EIP - 2015-01-11, HDFFR-1477

o  CMake fails to set the full path to the install location on Windows:
    The configuration file for examples, HDF4_Examples.cmake, must be updated
    with the correct value by editing the file or using the INSTALLDIR option.
    This issue is because of spaces in the path.
   ADB - 2014/02/03

o  CMake "make install" fails installing the tools:
    Use CPack to create an install package.
   ADB - 2014/02/03

o  When building on Linux/UNIX platforms, the szip shared library files must
   be in the system library path.  This can be done by adding a link to
   the libsz.* files in the /usr/lib folder or by adding the library
   location to the LD_LIBRARY_PATH environment variable.
      Ex. export LD_LIBRARY_PATH=path_to_szip_lib:$LD_LIBRARY_PATH
   Optionally, one can use the static szip library files by adding '-static'
   to the CFLAGS environment variable.

o  Existing data written by an HDF4 Library prior to HDF 4.2r2:
   When a one-dimensional SDS and a dimension scale have
   the same name, subsequent accesses to the dimension scale or to the
   SDS might produce undesired results because the libraries could not
   distinguish between the two objects.  In the case of writing, data
   might even be corrupted.  For example, SDS data might be written to a
   dimension variable or vice versa. (bugzilla #624)

   HDF4 Library Releases 4.2r2 and later make a distinction between an SDS
   and a dimension variable.  However, as with older versions, these recent
   versions are unable to detect such conflicts in files created by earlier
   releases.  It is therefore STRONGLY recommended to check for such name
   duplication before working with data created with a pre-4.2r2 library.

   The functions SDgetnumvars_byname and SDnametoindices are provided
   to help detect such name conflicts and select the correct object to
   access, respectively; see the HDF Reference Manual entries for
   further details.
   FB - 2009/01/26
   BMR - revised 2011/06/24

o  N-bit compression is not supported with Fortran APIs.

o  Using both fill-value and compression on SD datasets does not work.

o  When using PGI compilers, make sure that the JPEG library is also compiled
   with a PGI C compiler; linking with a JPEG library built with gcc causes
   JPEG library tests to fail.  To bypass the problem:

       x Set LIBS flag to $PGI_JPEG_INSTALL_DIR/lib/libjpeg.a
         where $PGI_JPEG_INSTALL_DIR points to the installation directory
         for the PGI-compiled JPEG library:

         setenv LIBS $PGI_JPEG_INSTALL_DIR/lib/libjpeg.a

       x Use the --with-jpeg=$PGI_JPEG_INSTALL_DIR configure flag to
         configure with the PGI-compiled JPEG library:

         ./configure --with-jpeg=$PGI_JPEG_INSTALL_DIR --with-zlib....

o  In order for the API SDgetdatasize to get the correct compressed size
   of the data, the dataset needs to be closed (SDendaccess) or read
   (SDreaddata) after being written and before SDgetdatasize is called.
   BMR - 2008/11/22

