subsys: fs/nvs: Rewrite for improved robustness

On flash NVS was stored one entry after another including the metadata
of each entry. This has the disadvantage that when an incomplete write
is performed (e.g. due to power failure) the complete sector had to be
rewritten to get a completely functional system.

The present rewrite changed the storage in flash of the data. For each
sector the data is now written as follows: the data itself at the
beginning of the sector (one after the other), the metadata (id, length,
data offset in the sector, and a crc of the metadata) is written from
the end of the sector. The metadata is of fixed size (8 byte) and for
a sector that is completely occupied a metadata entry of all zeros is
used.

Writing data to flash always is done by:
1. Writing the data,
2. Writing the metadata.

If an incomplete write is done NVS will ignore this incomplete write.

At the same time the following improvements were done:
1. NVS now support 65536 sectors of each 65536 byte.
2. The sector size no longer requires to be a power of 2 (but it
still needs to be a multiple of the flash erase page size).
3. NVS now also keeps track of the free space available.

Signed-off-by: Laczen JMS <laczenjms@gmail.com>

doc/subsystems/nvs/nvs.rst

@@ -10,56 +10,32 @@ new sector in the flash area is prepared for use (erased). Before erasing the
 sector it is checked that identifier - data pairs exist in the sectors in use,
 if not the id-data pair is copied.
 
-The id is a 16-bit unsigned number, where two values are reserved:
-
-- ``0xFFFF`` is used to determine free locations in flash
-- ``0x0000`` is used to determine jumps or to correct errors
-
-NVS ensures that for each id there is at least one id-data pair stored in flash
-at all time.
+The id is a 16-bit unsigned number. NVS ensures that for each used id there is
+at least one id-data pair stored in flash at all time.
 
 NVS allows storage of binary blobs, strings, integers, longs, and any
 combination of these.
 
-Each element is stored in flash as: ``length-data-id-length``. The length
-parameter is the size in byte of the data. It is added at the start and the end
-of a data block for two purposes:
+Each element is stored in flash as metadata (8 byte) and data. The metadata is
+written in a table starting from the end of a nvs sector, the data is
+written one after the other from the start of the sector. The metadata consists
+of: id, data offset in sector, data length, part (unused) and a crc.
 
-- It allows the verification of a data block, in case of a incomplete write
-  the length at the end is missing.
-- It allows NVS to travel forward and backward over the flash data blocks.
-
-
-To reduce flash usage the size of the length is dependent on the size of the
-data. For data size < 128 byte length occupies 1 byte in flash, for data size
->= 128 byte length occupies 2 bytes.
-
-If an error is detected in the flash storage (during writing) NVS is placed in
-a locked state and becomes a read-only storage system. This allows the user
-application to read all variables up to the error location to recover as much
-as possible. A call to nvs_reinit() when in locked state erases the flash
-storage and reopens NVS for storage.
-
-NVS will also be placed in a locked state when it is full meaning there is no
-extra space to store data. A call to nvs_reinit() when in locked state erases
-the flash  storage and reopens NVS for storage.
+A write of data to nvs always starts with writing the data, followed by a write
+of the metadata. Data that is written in flash without metadata is ignored
+during initialization.
 
 During initialization NVS will verify the data stored in flash, if it
-encounters an error it will correct the error and restore the data as much as
-possible. All data that is found before the error is restored.
+encounters an error it will ignore any data with missing/incorrect metadata.
 
-NVS has a configuration option to enable extra flash protection
-(``CONFIG_NVS_FLASH_PROTECTION=y``), when selected NVS does a extra check
-before writing data to flash. If the id-data pair is unchanged no write to
-flash is performed. The down-side of this protection is that NVS needs to read
-the storage system to check if the id-data pair is unchanged. If you are
-already performing such a check you can disable the extra flash protection
-(``CONFIG_NVS_FLASH_PROTECTION=n``).
-
-To protect the storage system against frequent sector erases the size of
-id-data pairs is limited to ``CONFIG_NVS_MAX_ELEM_SIZE``. This limit is
-by default set to 1/4 of the sector size.
+NVS checks the id-data pair before writing data to flash. If the id-data pair
+is unchanged no write to flash is performed.
 
+To protect the flash area against frequent erases it is important that there is
+sufficient free space. NVS has a protection mechanism to avoid getting in a
+endless loop of flash page erases when there is limited free space. When such
+an endless loop is detected NVS is placed in a locked state and becomes a
+read-only file system.
 
 For NVS the file system is declared as:
 
@@ -69,7 +45,6 @@ For NVS the file system is declared as:
 	.sector_size = NVS_SECTOR_SIZE,
 	.sector_count = NVS_SECTOR_COUNT,
 	.offset = NVS_STORAGE_OFFSET,
-	.max_len = NVS_MAX_ELEM_SIZE,
 	};
 
 where
@@ -79,7 +54,6 @@ where
 - ``NVS_SECTOR_COUNT`` is the number of sectors, it is at least 2, one
   sector is always kept empty to allow copying of existing data.
 - ``NVS_STORAGE_OFFSET`` is the offset of the storage area in flash.
-- ``NVS_MAX_ELEM_SIZE`` is the maximum item size.
 
 
 Flash wear
@@ -98,14 +72,14 @@ Suppose we use a 4 bytes state variable that is changed every minute and
 needs to be restored after reboot. NVS has been defined with a sector_size
 equal to the pagesize (1024 bytes) and 2 sectors have been defined.
 
-Each write of the state variable requires 8 bytes of flash storage (length: 1
-byte, id: 2 bytes, data: 4 bytes, length: 1 byte). When storing the data the
-first sector will be full after 1024/8 = 128 minutes. After another 128
+Each write of the state variable requires 12 bytes of flash storage: 8 bytes
+for the metadata and 4 bytes for the data. When storing the data the
+first sector will be full after 1024/12 = 85.33 minutes. After another 85.33
 minutes, the second sector is full.  When this happens, because we're using
 only two sectors, the first sector will be used for storage and will be erased
-after 256 minutes of system time.  With the expected device life of 20,000
-writes, with two sectors writing every 256 minutes, the device should last
-about 256 * 20,000 minutes, or about 9.75 years.
+after 171 minutes of system time.  With the expected device life of 20,000
+writes, with two sectors writing every 171 minutes, the device should last
+about 171 * 20,000 minutes, or about 6.5 years.
 
 More generally then, with
 
@@ -114,12 +88,7 @@ More generally then, with
 - ``SECTOR_SIZE`` in bytes, and
 - ``PAGE_ERASES`` as the number of times the page can be erased,
 
-for DS < 128 byte, the expected device life (in minutes) can be calculated as::
-
-   SECTOR_COUNT * SECTOR_SIZE * PAGE_ERASES / (NS * (DS+4)) minutes
-
-for DS >= 128 byte, the expected device life (in minutes) can be calculated
-as::
+the expected device life (in minutes) can be calculated as::
 
    SECTOR_COUNT * SECTOR_SIZE * PAGE_ERASES / (NS * (DS+8)) minutes
 

include/nvs/nvs.h

@@ -29,12 +29,12 @@ extern "C" {
 /**
  * @brief Non-volatile Storage File system structure
  *
- * @param write_location Next write location
- * @param entry_location Start of the filesystem
  * @param offset File system offset in flash
+ * @param ate_wra: Allocation table entry write address. Addresses are stored
+ * as u32_t: high 2 bytes are sector, low 2 bytes are offset in sector,
+ * @param data_wra: Data write address.
  * @param sector_size File system is divided into sectors each sector should be
- * multiple of pagesize and also a power of 2
- * @param max_len Maximum size of storage items
+ * multiple of pagesize
  * @param sector_count Amount of sectors in the file systems
  * @param write_block_size Alignment size
  * @param locked State of the filesystem, locked = true means the filesystem is
@@ -43,15 +43,14 @@ extern "C" {
  * @param flash_device Flash Device
  */
 struct nvs_fs {
-	off_t write_location; /* next write location */
-	off_t entry_location; /* start of the filesystem */
-	off_t offset; /* filesystem offset in flash */
-	u16_t sector_size; /* filesystem is divided into sectors,
-			    * sector size should be multiple of pagesize
-			    * and a power of 2
-			    */
-	u16_t max_len; /* maximum size of stored item, set to sector_size/4 */
-	u8_t sector_count; /* how many sectors in the filesystem */
+	off_t offset;		/* filesystem offset in flash */
+	u32_t ate_wra;		/* next alloc table entry write address */
+	u32_t data_wra;		/* next data write address */
+	u32_t free_space;	/* free space available in file system */
+	u16_t sector_size;	/* filesystem is divided into sectors,
+				 * sector size should be multiple of pagesize
+				 */
+	u16_t sector_count;	/* amount of sectors in the filesystem */
 
 	u8_t write_block_size; /* write block size for alignment */
 	bool locked; /* the filesystem is locked after an error occurred
@@ -119,8 +118,7 @@ int nvs_clear(struct nvs_fs *fs);
  * @param len Number of bytes to be written
  *
  * @return Number of bytes written. On success, it will be equal to the number
- * of bytes requested to be written. Any other value, indicates an error. Will
- * return -ERRNO code on error.
+ * of bytes requested to be written. On error returns -ERRNO code.
  */
 
 ssize_t nvs_write(struct nvs_fs *fs, u16_t id, const void *data, size_t len);
@@ -148,10 +146,9 @@ int nvs_delete(struct nvs_fs *fs, u16_t id);
  * @param len Number of bytes to be read
  *
  * @return Number of bytes read. On success, it will be equal to the number
- * of bytes requested to be read. Any other value, indicates an error. When
- * the number of bytes read is larger than the number of bytes requested to
- * read this indicates not all bytes were read, and more data is available.
- * Return -ERRNO code on error.
+ * of bytes requested to be read. When the return value is larger than the
+ * number of bytes requested to read this indicates not all bytes were read,
+ * and more data is available. On error returns -ERRNO code.
  */
 ssize_t nvs_read(struct nvs_fs *fs, u16_t id, void *data, size_t len);
 
@@ -167,10 +164,9 @@ ssize_t nvs_read(struct nvs_fs *fs, u16_t id, void *data, size_t len);
  * @param cnt History counter: 0: latest entry, 1:one before latest ...
  *
  * @return Number of bytes read. On success, it will be equal to the number
- * of bytes requested to be read. Any other value, indicates an error. When
- * the number of bytes read is larger than the number of bytes requested to
- * read this indicates not all bytes were read, and more data is available.
- * Return -ERRNO code on error.
+ * of bytes requested to be read. When the return value is larger than the
+ * number of bytes requested to read this indicates not all bytes were read,
+ * and more data is available. On error returns -ERRNO code.
  */
 ssize_t nvs_read_hist(struct nvs_fs *fs, u16_t id, void *data, size_t len,
 		  u16_t cnt);

samples/subsys/nvs/prj.conf

@@ -3,5 +3,4 @@ CONFIG_FLASH=y
 CONFIG_NVS=y
 CONFIG_NVS_LOG=y
 CONFIG_NVS_LOG_LEVEL=4
-CONFIG_NVS_PROTECT_FLASH=y
 CONFIG_REBOOT=y

samples/subsys/nvs/src/main.c

@@ -50,13 +50,10 @@
 #define NVS_STORAGE_OFFSET FLASH_AREA_STORAGE_OFFSET /* Start address of the
 						      * filesystem in flash
 						      */
-#define NVS_MAX_ELEM_SIZE 256 /* Largest item that can be stored */
-
 static struct nvs_fs fs = {
 	.sector_size = NVS_SECTOR_SIZE,
 	.sector_count = NVS_SECTOR_COUNT,
 	.offset = NVS_STORAGE_OFFSET,
-	.max_len = NVS_MAX_ELEM_SIZE,
 };
 
 /* 1000 msec = 1 sec */
@@ -71,6 +68,7 @@ static struct nvs_fs fs = {
 #define STRING_ID 4
 #define LONG_ID 5
 
+
 void main(void)
 {
 	int rc = 0, cnt = 0, cnt_his = 0;
@@ -176,8 +174,6 @@ void main(void)
 		}
 	}
 
-
-
 	cnt = 5;
 	while (1) {
 		k_sleep(SLEEP_TIME);

subsys/fs/nvs/nvs_priv.h

@@ -15,20 +15,31 @@ extern "C" {
 #define SYS_LOG_LEVEL CONFIG_NVS_LOG_LEVEL
 #include <logging/sys_log.h>
 
-#define NVS_MAX_LEN 0x7ffd
-
 /*
- * Special id values
+ * MASKS AND SHIFT FOR ADDRESSES
+ * an address in nvs is an u32_t where:
+ *   high 2 bytes represent the sector number
+ *   low 2 bytes represent the offset in a sector
  */
-#define NVS_ID_EMPTY		0xFFFF
-#define NVS_ID_JUMP		0x0000
+#define ADDR_SECT_MASK 0xFFFF0000
+#define ADDR_SECT_SHIFT 16
+#define ADDR_OFFS_MASK 0x0000FFFF
 
 /*
  * Status return values
  */
 #define NVS_STATUS_NOSPACE 1
 
-#define NVS_MIN_WRITE_BLOCK_SIZE 4
+#define NVS_BLOCK_SIZE 8
+
+/* Allocation Table Entry */
+struct nvs_ate {
+	u16_t id;	/* data id */
+	u16_t offset;	/* data offset in sector */
+	u16_t len;	/* data len in sector */
+	u8_t part;	/* part of a multipart data - future extension */
+	u8_t crc8;	/* crc8 check of the entry */
+};
 
 #ifdef __cplusplus
 }