NiFi 1.11 - High repository storage usage

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

NiFi 1.11 - High repository storage usage

Valentina Ivanova
Hello!

I see (screenshot attached) quite high (86%) storage usage for all three Flow File, Content & Provenance Repositories in the System Diagnostics. When I check the size of the respective folders on the disk they don't even add up to 12,24 GB shown on the screenshot.
I also have the following settings for the content repository in nifi.properties:
nifi.content.repository.archive.max.retention.period=12 hours
nifi.content.repository.archive.max.usage.percentage=50%          
nifi.content.repository.archive.enabled=true
nifi.content.repository.always.sync=false


So I am wondering where these numbers are coming from and how to reduce the storage utilization (I assume such high utilization will impact the performance)? Why is the content repository usage 86% even though the max.usage.percentage is set at 50%.

Many thanks in advance & all the best

Valentina




system-diagnostics.png (24K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: NiFi 1.11 - High repository storage usage

Wesley C. Dias de Oliveira
Hi, Valentina.

I've experienced the same issue on an old installation.

In my case, it was related to memory usage.

The system begins to dump things to the disk when there's no available memory.

Have you checked the memory( params?

# JVM memory settings
java.arg.2=-Xms512m
java.arg.3=-Xmx512m






Em qui., 25 de jun. de 2020 às 09:50, Valentina Ivanova <[hidden email]> escreveu:
Hello!

I see (screenshot attached) quite high (86%) storage usage for all three Flow File, Content & Provenance Repositories in the System Diagnostics. When I check the size of the respective folders on the disk they don't even add up to 12,24 GB shown on the screenshot.
I also have the following settings for the content repository in nifi.properties:
nifi.content.repository.archive.max.retention.period=12 hours
nifi.content.repository.archive.max.usage.percentage=50%          
nifi.content.repository.archive.enabled=true
nifi.content.repository.always.sync=false


So I am wondering where these numbers are coming from and how to reduce the storage utilization (I assume such high utilization will impact the performance)? Why is the content repository usage 86% even though the max.usage.percentage is set at 50%.

Many thanks in advance & all the best

Valentina





--
Grato,
Wesley C. Dias de Oliveira.

Linux User nº 576838.
Reply | Threaded
Open this post in threaded view
|

AW: NiFi 1.11 - High repository storage usage

Dobbernack, Harald (Key-Work)
In reply to this post by Valentina Ivanova

Hi,

 

presumably all three respositories are on the same partition or drive? I believe the screenshot view you posted shows what the OS reports as the space usage of the whole partition/drive on which the repositories are sitting. Best practice would be to place each repository on its own partition.

 

Greetings

Harald

 

Von: Valentina Ivanova <[hidden email]>
Gesendet: Donnerstag, 25. Juni 2020 14:50
An: [hidden email]
Betreff: NiFi 1.11 - High repository storage usage

 

Hello!

 

I see (screenshot attached) quite high (86%) storage usage for all three Flow File, Content & Provenance Repositories in the System Diagnostics. When I check the size of the respective folders on the disk they don't even add up to 12,24 GB shown on the screenshot.

I also have the following settings for the content repository in nifi.properties:

nifi.content.repository.archive.max.retention.period=12 hours

nifi.content.repository.archive.max.usage.percentage=50%          

nifi.content.repository.archive.enabled=true

nifi.content.repository.always.sync=false

 

 

So I am wondering where these numbers are coming from and how to reduce the storage utilization (I assume such high utilization will impact the performance)? Why is the content repository usage 86% even though the max.usage.percentage is set at 50%.

 

Many thanks in advance & all the best

 

Valentina

 

 

 

 
 
Harald Dobbernack

Key-Work Consulting GmbH | Kriegsstr. 100 | 76133 | Karlsruhe | Germany | www.key-work.de | Datenschutz
Fon: +49-721-78203-264 | E-Mail: [hidden email] | Fax: +49-721-78203-10
 
Key-Work Consulting GmbH, Karlsruhe, HRB 108695, HRG Mannheim
Geschäftsführer: Andreas Stappert, Tobin Wotring

Reply | Threaded
Open this post in threaded view
|

AW: NiFi 1.11 - High repository storage usage

Dobbernack, Harald (Key-Work)
  • When I check the size of the respective folders on the disk they don't even add up to 12,24 GB shown on the screenshot.

 

I’m guessing you have other stuff on the drive or partition as well and not only the repositories.

 

Von: Dobbernack, Harald (Key-Work)
Gesendet: Donnerstag, 25. Juni 2020 14:58
An: [hidden email]
Betreff: AW: NiFi 1.11 - High repository storage usage

 

Hi,

 

presumably all three respositories are on the same partition or drive? I believe the screenshot view you posted shows what the OS reports as the space usage of the whole partition/drive on which the repositories are sitting. Best practice would be to place each repository on its own partition.

 

Greetings

Harald

 

Von: Valentina Ivanova <[hidden email]>
Gesendet: Donnerstag, 25. Juni 2020 14:50
An: [hidden email]
Betreff: NiFi 1.11 - High repository storage usage

 

Hello!

 

I see (screenshot attached) quite high (86%) storage usage for all three Flow File, Content & Provenance Repositories in the System Diagnostics. When I check the size of the respective folders on the disk they don't even add up to 12,24 GB shown on the screenshot.

I also have the following settings for the content repository in nifi.properties:

nifi.content.repository.archive.max.retention.period=12 hours

nifi.content.repository.archive.max.usage.percentage=50%          

nifi.content.repository.archive.enabled=true

nifi.content.repository.always.sync=false

 

 

So I am wondering where these numbers are coming from and how to reduce the storage utilization (I assume such high utilization will impact the performance)? Why is the content repository usage 86% even though the max.usage.percentage is set at 50%.

 

Many thanks in advance & all the best

 

Valentina

 

 

 

 
 
Harald Dobbernack

Key-Work Consulting GmbH | Kriegsstr. 100 | 76133 | Karlsruhe | Germany | www.key-work.de | Datenschutz
Fon: +49-721-78203-264 | E-Mail: [hidden email] | Fax: +49-721-78203-10
 
Key-Work Consulting GmbH, Karlsruhe, HRB 108695, HRG Mannheim
Geschäftsführer: Andreas Stappert, Tobin Wotring

Reply | Threaded
Open this post in threaded view
|

Re: NiFi 1.11 - High repository storage usage

Valentina Ivanova
Hi Wesley & Harald,

Thanks for the quick replies!

[hidden email] I have recently increased these settings due to other reasons, so that might be the primary cause.

@Harald You are right, they are on the same drive. I assumed that usage is relative to the nifi usage/settings not the OS. I will now change so each repository gets its own partition.

Thanks!

Valentina

From: Dobbernack, Harald (Key-Work) <[hidden email]>
Sent: Thursday, 25 June 2020 15:08
To: [hidden email] <[hidden email]>
Subject: AW: NiFi 1.11 - High repository storage usage
 
  • When I check the size of the respective folders on the disk they don't even add up to 12,24 GB shown on the screenshot.

 

I’m guessing you have other stuff on the drive or partition as well and not only the repositories.

 

Von: Dobbernack, Harald (Key-Work)
Gesendet: Donnerstag, 25. Juni 2020 14:58
An: [hidden email]
Betreff: AW: NiFi 1.11 - High repository storage usage

 

Hi,

 

presumably all three respositories are on the same partition or drive? I believe the screenshot view you posted shows what the OS reports as the space usage of the whole partition/drive on which the repositories are sitting. Best practice would be to place each repository on its own partition.

 

Greetings

Harald

 

Von: Valentina Ivanova <[hidden email]>
Gesendet: Donnerstag, 25. Juni 2020 14:50
An: [hidden email]
Betreff: NiFi 1.11 - High repository storage usage

 

Hello!

 

I see (screenshot attached) quite high (86%) storage usage for all three Flow File, Content & Provenance Repositories in the System Diagnostics. When I check the size of the respective folders on the disk they don't even add up to 12,24 GB shown on the screenshot.

I also have the following settings for the content repository in nifi.properties:

nifi.content.repository.archive.max.retention.period=12 hours

nifi.content.repository.archive.max.usage.percentage=50%          

nifi.content.repository.archive.enabled=true

nifi.content.repository.always.sync=false

 

 

So I am wondering where these numbers are coming from and how to reduce the storage utilization (I assume such high utilization will impact the performance)? Why is the content repository usage 86% even though the max.usage.percentage is set at 50%.

 

Many thanks in advance & all the best

 

Valentina

 

 

 

 
 
Harald Dobbernack

Key-Work Consulting GmbH | Kriegsstr. 100 | 76133 | Karlsruhe | Germany | www.key-work.de | Datenschutz
Fon: +49-721-78203-264 | E-Mail: [hidden email] | Fax: +49-721-78203-10
 
Key-Work Consulting GmbH, Karlsruhe, HRB 108695, HRG Mannheim
Geschäftsführer: Andreas Stappert, Tobin Wotring