NiFi Docker

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

NiFi Docker

ddewaele
Hi,

We're using Nifi in a containerized environment (using docker-compose for local development and test, and docker swarm in production).

The current docker image doesn't have a lot of environment options, meaning if you want to run Nifi with some custom config like the one below :

    environment:
      - NIFI_WEB_HTTP_HOST=xxx
      - NIFI_CLUSTER_HOST_NAME=xxx
      - NIFI_CLUSTER_HOST_PORT=xxx
      - NIFI_CLUSTER_PORT=xxx
      - NIFI_ZOOKEEPER_CONNECT_STRING=xxx
      - NIFI_CONTENT_REPOSITORY_ARCHIVE_MAX_USAGE_PERCENTAGE=xxx
      - CONFIG_URI_LABEL=xxx

You need to create your own "wrapper" docker image (extending the base apache nifi one).

Stuff that we typically to in a nifi installation is

- change values in nifi.properties (a generic solution could be created for that where nifi property keys could be provided as environment variables to the docker container)
- install custom processors (nar files)
- custom log configuration (logback.xml)
- custom bootstrap values
- copy flow templates in nifi so they can be used immediately after startup.

Have you ever thought of extending the current docker image, to allow a little bit more customization like this ones above.

Some other things I noticed :

- the readme on https://hub.docker.com/r/apache/nifi/ doesn't mention docker usage it all. seems to be the default nifi readme.

- The current docker image contains 2 layers of 1gb each. I'm guessing this is the result of a) downloading and untarring the nifi distribution and b) executing the chown command. Is there a reason these are spread out over 2 docker run commands ?

f5f88f68e088: Downloading [=>                                                 ]   25.9MB/978MB
5ed4b763cde4: Downloading [=>                                                 ]  34.02MB/978MB
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: NiFi Docker

Aldrin Piri
Hi Dave,

Yes! Most certainly to all these things you are suggesting.

Would you be interested in opening a JIRA to capture these?  If not, I will be sure to do so.  Definitely has been an item to address moving forward as you are a limited with the current image.  We may not be able to support every configuration but we should strive to get an image that covers all the base modes of operation.  The items you highlight are well within that purview.

As far as the README, given how the images are published through ASF channels, we do not have direct access to the Docker hub configuration.  I believe there is a way we can make it reflect a custom README in that folder, but would have to do some research/experimentation.

As far as the large layers, you would be correct and this is something that has been remedied code wise via https://issues.apache.org/jira/browse/NIFI-4057.



On Sun, Jul 30, 2017 at 6:32 AM, ddewaele <[hidden email]> wrote:
Hi,

We're using Nifi in a containerized environment (using docker-compose for
local development and test, and docker swarm in production).

The current docker image doesn't have a lot of environment options, meaning
if you want to run Nifi with some custom config like the one below :

    environment:
      - NIFI_WEB_HTTP_HOST=xxx
      - NIFI_CLUSTER_HOST_NAME=xxx
      - NIFI_CLUSTER_HOST_PORT=xxx
      - NIFI_CLUSTER_PORT=xxx
      - NIFI_ZOOKEEPER_CONNECT_STRING=xxx
      - NIFI_CONTENT_REPOSITORY_ARCHIVE_MAX_USAGE_PERCENTAGE=xxx
      - CONFIG_URI_LABEL=xxx

You need to create your own "wrapper" docker image (extending the base
apache nifi one).

Stuff that we typically to in a nifi installation is

- change values in nifi.properties (a generic solution could be created for
that where nifi property keys could be provided as environment variables to
the docker container)
- install custom processors (nar files)
- custom log configuration (logback.xml)
- custom bootstrap values
- copy flow templates in nifi so they can be used immediately after startup.

Have you ever thought of extending the current docker image, to allow a
little bit more customization like this ones above.

Some other things I noticed :

- the readme on https://hub.docker.com/r/apache/nifi/ doesn't mention docker
usage it all. seems to be the default nifi readme.

- The current docker image contains 2 layers of 1gb each. I'm guessing this
is the result of a) downloading and untarring the nifi distribution and b)
executing the chown command. Is there a reason these are spread out over 2
docker run commands ?

f5f88f68e088: Downloading [=>
]   25.9MB/978MB
5ed4b763cde4: Downloading [=>
]  34.02MB/978MB



--
View this message in context: http://apache-nifi-users-list.2361937.n4.nabble.com/NiFi-Docker-tp2562.html
Sent from the Apache NiFi Users List mailing list archive at Nabble.com.

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: NiFi Docker

Jeremy Dyer
Dave,

The echo Aldrin's comment I think this would be fantastic! Once you have a JIRA open I'd be more than happy to work with you and help get your suggested updates into the the code base. 

Thanks,
Jeremy Dyer

On Mon, Jul 31, 2017 at 8:40 AM, Aldrin Piri <[hidden email]> wrote:
Hi Dave,

Yes! Most certainly to all these things you are suggesting.

Would you be interested in opening a JIRA to capture these?  If not, I will be sure to do so.  Definitely has been an item to address moving forward as you are a limited with the current image.  We may not be able to support every configuration but we should strive to get an image that covers all the base modes of operation.  The items you highlight are well within that purview.

As far as the README, given how the images are published through ASF channels, we do not have direct access to the Docker hub configuration.  I believe there is a way we can make it reflect a custom README in that folder, but would have to do some research/experimentation.

As far as the large layers, you would be correct and this is something that has been remedied code wise via https://issues.apache.org/jira/browse/NIFI-4057.



On Sun, Jul 30, 2017 at 6:32 AM, ddewaele <[hidden email]> wrote:
Hi,

We're using Nifi in a containerized environment (using docker-compose for
local development and test, and docker swarm in production).

The current docker image doesn't have a lot of environment options, meaning
if you want to run Nifi with some custom config like the one below :

    environment:
      - NIFI_WEB_HTTP_HOST=xxx
      - NIFI_CLUSTER_HOST_NAME=xxx
      - NIFI_CLUSTER_HOST_PORT=xxx
      - NIFI_CLUSTER_PORT=xxx
      - NIFI_ZOOKEEPER_CONNECT_STRING=xxx
      - NIFI_CONTENT_REPOSITORY_ARCHIVE_MAX_USAGE_PERCENTAGE=xxx
      - CONFIG_URI_LABEL=xxx

You need to create your own "wrapper" docker image (extending the base
apache nifi one).

Stuff that we typically to in a nifi installation is

- change values in nifi.properties (a generic solution could be created for
that where nifi property keys could be provided as environment variables to
the docker container)
- install custom processors (nar files)
- custom log configuration (logback.xml)
- custom bootstrap values
- copy flow templates in nifi so they can be used immediately after startup.

Have you ever thought of extending the current docker image, to allow a
little bit more customization like this ones above.

Some other things I noticed :

- the readme on https://hub.docker.com/r/apache/nifi/ doesn't mention docker
usage it all. seems to be the default nifi readme.

- The current docker image contains 2 layers of 1gb each. I'm guessing this
is the result of a) downloading and untarring the nifi distribution and b)
executing the chown command. Is there a reason these are spread out over 2
docker run commands ?

f5f88f68e088: Downloading [=>
]   25.9MB/978MB
5ed4b763cde4: Downloading [=>
]  34.02MB/978MB



--
View this message in context: http://apache-nifi-users-list.2361937.n4.nabble.com/NiFi-Docker-tp2562.html
Sent from the Apache NiFi Users List mailing list archive at Nabble.com.


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: NiFi Docker

ddewaele
Great .... also have some ideas about this. I'll log a JIRA and elaborate on those.

We can then see on how to move this forward. (willing to do a pull request for this).
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: NiFi Docker

Aldrin Piri
Sounds great.  Will keep an eye out for it.  Thanks!

On Tue, Aug 1, 2017 at 10:50 AM, ddewaele <[hidden email]> wrote:
Great .... also have some ideas about this. I'll log a JIRA and elaborate on
those.

We can then see on how to move this forward. (willing to do a pull request
for this).



--
View this message in context: http://apache-nifi-users-list.2361937.n4.nabble.com/NiFi-Docker-tp2562p2576.html
Sent from the Apache NiFi Users List mailing list archive at Nabble.com.

Loading...