Jupyter, Python, Oracle and Docker Part 2

You can find the Python Jupyter Notebook used in this exercise here.

OracleDockerImage

Build Docker Oracle Database Base Image

The following notebook goes through the process of building an Oracle Docker image of the Oracle Database. If you are just wanting to get the Oracle Database running inside of Docker I strongly suggest that you use the docker files and guides in the Oracle Github repository. The approach documented below is very much for someone who is interested in a high level of control over the various steps in the installation and configuration of the Oracle Database or simply to understand how the various teps work.

Prerequisites

The process documented below uses a Jupyter Notebook (iPython). The reason I use this approach and not straight python is that it's easy to change and is self documenting. It only takes a few minutes to set up the environment. I've included a requirements file which makes it very simple to install the needed Python libraries. I go through the process of setting up a Jupyter environment for Mac and Linux here.

Running the notebook

Typically the only modification that the user will need to do is to modifythe values in the "Parameters" section. The code can then be run by pressing "Command SHIFT" on a Mac or "Ctrl Shift" on Windows. Or by pressing the "Play" icon in the tool bar. It is also possible to run all of the cells automatically, you can do this from "Run" menu item.

In [ ]:
import docker
import os
import tarfile
from prettytable import PrettyTable
from IPython.display import HTML, display, Markdown
import humanize
import re
from ipynb.fs.full.OracleDockerDatabaseFunctions import list_images,list_containers,copy_to,create_and_run_script,containter_exec,copy_string_to_file,containter_root_exec

client = docker.from_env(timeout=600)

Parameters

This section details the parameters used to define the docker image you'll end up creating. It's almost certainly the case that you'll need to change the parameters in the first section. The parameters in the second section can be changed if there's something i.e. hostname that you want to change

In [ ]:
# You'll need to to change the following two parameters to reflect your environment
host_orabase = '/Users/dgiles/oradata18c' # The directory on the host where you'll stage the persisted datafiles
host_install_dir = '/Users/dgiles/Downloads/oracle18_software' # The directory on the host where the downloaded Oracle Database zip file is.
# You can change any of the following parameters but it's not necssary
p_host_name = 'oracle_db'
oracle_version = '18.0.0'
oracle_base = '/u01/app/oracle'
oracle_home = f'{oracle_base}/product/{oracle_version}/dbhome_1'
db_name = 'ORCL'
oracle_sid= db_name
path=f'{oracle_home}/bin:$PATH'
tns_admin=f'{oracle_base}/oradata/dbconfig'
container_oradata = '/u01/app/oracle/oradata'
r_area = f'{oracle_base}/oradata/recovery_area'
a_area = f'{oracle_base}/admin/ORCL/adump'
container_install_dir = '/u01/install_files'
path = f'{oracle_home}/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'

Attempt to create the needed directories on the host and print warnings if needed

In [42]:
try:
    os.makedirs(host_orabase, exist_ok=True)
    os.makedirs(host_install_dir, exist_ok=True)
except:
   display(Markdown(f'**WARNING** : Unable to create directories {host_orabase} and {host_install_dir}'))
    
files = os.listdir(host_install_dir)
found_similar:bool = False
for file in files:
    if file.startswith('LINUX.X64'):
        found_similar = True
        break
if not found_similar:
    display(Markdown(f"**WARNING** : Are you sure that you've downloaded the needed Oracle executable to the `{host_install_dir}` directory"))

The first step in creating a usable image is to create a docker file. This details what the docker container will be based on and what needs to be installed. It will use the parameters defined above. It does require network connectivity for this to work as docker will pull down the required images and RPMs.

In [43]:
script = f'''
FROM oraclelinux:7-slim

ENV ORACLE_BASE={oracle_base} \
    ORACLE_HOME={oracle_home} \
    ORACLE_SID={oracle_sid} \
    PATH={oracle_home}/bin:$PATH
    
RUN yum -y install unzip
RUN yum -y install oracle-database-preinstall-18c
RUN yum -y install openssl
    
# RUN groupadd -g 500 dba
# RUN useradd -ms /bin/bash -g dba oracle

RUN mkdir -p $ORACLE_BASE
RUN mkdir -p $ORACLE_HOME
RUN mkdir -p {container_install_dir}

RUN chown -R oracle:dba {oracle_base}
RUN chown -R oracle:dba {oracle_home}
RUN chown -R oracle:dba {container_install_dir}

USER oracle
WORKDIR /home/oracle

VOLUME ["$ORACLE_BASE/oradata"]
VOLUME ["{container_install_dir}"]
EXPOSE 1521 8080 5500
'''

with open('Dockerfile','w') as f:
    f.write(script)

And now we can create the image. The period of time for this operation to complete will depend on what docker images have already been downloaded/cached and your network speed.

In [44]:
image, output = client.images.build(path=os.getcwd(),dockerfile='Dockerfile', tag=f"linux_for_db{oracle_version}",rm="True",nocache="False")
for out in output:
    print(out)
{'stream': 'Step 1/16 : FROM oraclelinux:7-slim'}
{'stream': '\n'}
{'stream': ' ---> c3d869388183\n'}
{'stream': 'Step 2/16 : ENV ORACLE_BASE=/u01/app/oracle     ORACLE_HOME=/u01/app/oracle/product/18.0.0/dbhome_1     ORACLE_SID=ORCL     PATH=/u01/app/oracle/product/18.0.0/dbhome_1/bin:$PATH'}
{'stream': '\n'}
{'stream': ' ---> Running in f7be15a38f17\n'}
{'stream': 'Removing intermediate container f7be15a38f17\n'}
{'stream': ' ---> ec34e0a4ca0b\n'}
{'stream': 'Step 3/16 : RUN yum -y install unzip'}
{'stream': '\n'}
{'stream': ' ---> Running in b8ae90884550\n'}
{'stream': 'Loaded plugins: ovl\n'}
{'stream': 'Resolving Dependencies\n'}
{'stream': '--> Running transaction check\n'}
{'stream': '---> Package unzip.x86_64 0:6.0-19.el7 will be installed\n'}
{'stream': '--> Finished Dependency Resolution\n'}
{'stream': '\nDependencies Resolved\n'}
{'stream': '\n================================================================================\n Package        Arch            Version               Repository           Size\n================================================================================\nInstalling:\n unzip          x86_64          6.0-19.el7            ol7_latest          169 k\n\nTransaction Summary\n================================================================================\nInstall  1 Package\n\n'}
{'stream': 'Total download size: 169 k\n'}
{'stream': 'Installed size: 365 k\n'}
{'stream': 'Downloading packages:\n'}
{'stream': 'Running transaction check\n'}
{'stream': 'Running transaction test\n'}
{'stream': 'Transaction test succeeded\n'}
{'stream': 'Running transaction\n'}
{'stream': '  Installing : unzip-6.0-19.el7.x86_64                                      1/1'}
{'stream': ' \n  Verifying  : unzip-6.0-19.el7.x86_64                                      1/1'}
{'stream': ' \n\nInstalled:\n  unzip.x86_64 0:6.0-19.el7                                                     \n\n'}
{'stream': 'Complete!\n'}
{'stream': 'Removing intermediate container b8ae90884550\n'}
{'stream': ' ---> d7f9d5678e7f\n'}
{'stream': 'Step 4/16 : RUN yum -y install oracle-database-preinstall-18c'}
{'stream': '\n'}
{'stream': ' ---> Running in 35e010d92729\n'}
{'stream': 'Loaded plugins: ovl\n'}
{'stream': 'Resolving Dependencies\n'}
{'stream': '--> Running transaction check\n'}
{'stream': '---> Package oracle-database-preinstall-18c.x86_64 0:1.0-1.el7 will be installed\n'}
{'stream': '--> Processing Dependency: binutils for package: oracle-database-preinstall-18c-1.0-1.el7.x86_64\n'}
{'stream': '--> Processing Dependency: bc for package: oracle-database-preinstall-18c-1.0-1.el7.x86_64\n'}
{'stream': '--> Processing Dependency: initscripts for package: oracle-database-preinstall-18c-1.0-1.el7.x86_64\n'}
{'stream': '--> Processing Dependency: sysstat for package: oracle-database-preinstall-18c-1.0-1.el7.x86_64\n'}
{'stream': '--> Processing Dependency: xorg-x11-utils for package: oracle-database-preinstall-18c-1.0-1.el7.x86_64\n'}
{'stream': '--> Processing Dependency: bind-utils for package: oracle-database-preinstall-18c-1.0-1.el7.x86_64\n'}
{'stream': '--> Processing Dependency: net-tools for package: oracle-database-preinstall-18c-1.0-1.el7.x86_64\n'}
{'stream': '--> Processing Dependency: pam for package: oracle-database-preinstall-18c-1.0-1.el7.x86_64\n'}
{'stream': '--> Processing Dependency: compat-libcap1 for package: oracle-database-preinstall-18c-1.0-1.el7.x86_64\n'}
{'stream': '--> Processing Dependency: ksh for package: oracle-database-preinstall-18c-1.0-1.el7.x86_64\n'}
{'stream': '--> Processing Dependency: libaio-devel for package: oracle-database-preinstall-18c-1.0-1.el7.x86_64\n'}
{'stream': '--> Processing Dependency: xorg-x11-xauth for package: oracle-database-preinstall-18c-1.0-1.el7.x86_64\n'}
{'stream': '--> Processing Dependency: util-linux-ng for package: oracle-database-preinstall-18c-1.0-1.el7.x86_64\n'}
{'stream': '--> Processing Dependency: libaio for package: oracle-database-preinstall-18c-1.0-1.el7.x86_64\n'}
{'stream': '--> Processing Dependency: psmisc for package: oracle-database-preinstall-18c-1.0-1.el7.x86_64\n'}
{'stream': '--> Processing Dependency: module-init-tools for package: oracle-database-preinstall-18c-1.0-1.el7.x86_64\n'}
{'stream': '--> Processing Dependency: glibc-devel for package: oracle-database-preinstall-18c-1.0-1.el7.x86_64\n'}
{'stream': '--> Processing Dependency: compat-libstdc++-33 for package: oracle-database-preinstall-18c-1.0-1.el7.x86_64\n'}
{'stream': '--> Processing Dependency: libstdc++-devel for package: oracle-database-preinstall-18c-1.0-1.el7.x86_64\n'}
{'stream': '--> Processing Dependency: ethtool for package: oracle-database-preinstall-18c-1.0-1.el7.x86_64\n'}

....
....
....


{'stream': ' ---> Running in 7a359fa6a8d3\n'}
{'stream': 'Removing intermediate container 7a359fa6a8d3\n'}
{'stream': ' ---> b04ec8faaa7b\n'}
{'stream': 'Step 13/16 : WORKDIR /home/oracle'}
{'stream': '\n'}
{'stream': ' ---> Running in 6f3186c4b034\n'}
{'stream': 'Removing intermediate container 6f3186c4b034\n'}
{'stream': ' ---> db70e7f386dd\n'}
{'stream': 'Step 14/16 : VOLUME ["$ORACLE_BASE/oradata"]'}
{'stream': '\n'}
{'stream': ' ---> Running in a3807026181b\n'}
{'stream': 'Removing intermediate container a3807026181b\n'}
{'stream': ' ---> fa4f83782cac\n'}
{'stream': 'Step 15/16 : VOLUME ["/u01/install_files"]'}
{'stream': '\n'}
{'stream': ' ---> Running in 4a5210ddaf73\n'}
{'stream': 'Removing intermediate container 4a5210ddaf73\n'}
{'stream': ' ---> 74a6329f5a96\n'}
{'stream': 'Step 16/16 : EXPOSE 1521 8080 5500'}
{'stream': '\n'}
{'stream': ' ---> Running in 2226a299f10c\n'}
{'stream': 'Removing intermediate container 2226a299f10c\n'}
{'stream': ' ---> b350079361f4\n'}
{'aux': {'ID': 'sha256:b350079361f42e64ee37dbc5c8257a6734fc3fdd3c66ab500fc86a0adf3b93b7'}}
{'stream': 'Successfully built b350079361f4\n'}
{'stream': 'Successfully tagged linux_for_db18.0.0:latest\n'}

Once the image has been created we can start a container based on it.

In [45]:
db_container = client.containers.create(image.short_id,
                                       command="/bin/bash",
                                       hostname=p_host_name,
                                       tty=True,
                                       stdin_open=True,
                                       auto_remove=False,
                                       name=p_host_name,
                                       shm_size='3G',
                                       # network_mode='host',
                                       ports={1521:1522,5500:5501},
                                       volumes={host_orabase: {'bind': container_oradata, 'mode': 'rw'}, host_install_dir: {'bind': container_install_dir, 'mode': 'rw'}},
                                       environment={'PATH':path,'ORACLE_SID': db_name, 'ORACLE_BASE': oracle_base,'TNS_ADMIN': tns_admin, 'ORACLE_HOME':oracle_home}
                                       )

#                                        volumes={host_orabase: {'bind': oracle_base, 'mode': 'rw'}, host_install_dir: {'bind': container_install_dir, 'mode': 'rw'}},

db_container.start()
p_ip_adress = db_container.attrs['NetworkSettings']['IPAddress']

And then created the needed directory structure within it.

In [46]:
containter_exec(db_container, f'mkdir -p {container_oradata}/{db_name}')
containter_exec(db_container, f'mkdir -p {tns_admin}')
containter_exec(db_container, f'mkdir -p {r_area}/{db_name}')
containter_exec(db_container, f'mkdir -p {a_area}')
containter_exec(db_container, f'mkdir -p {oracle_base}/oraInventory')
containter_exec(db_container, f'mkdir -p {oracle_home}')
containter_root_exec(db_container,'usermod -a -G dba oracle')

Unzip Oracle Database software and validate

We now need to unzip the Oracle software which should be located in the host_install_dir variable. This is unzipped within the container not the host. NOTE: I don't stream the output because it's realtively large. It should take 2-5 minutes.

In [47]:
files = [f for f in os.listdir(host_install_dir) if f.endswith('.zip')]
    
if files == 0:
    display(Markdown(f"**There doesn't appear to be any zip files in the {host_install_dir} directory. This should contain the oracle database for Linux 64bit in its orginal zipped format**"))
else:
    display(Markdown(f'Unzipping `{files[0]}`'))
    containter_exec(db_container, f'unzip -o {container_install_dir}/{files[0]} -d {oracle_home}', show_output=False, stream_val=False)
    

Unzipping LINUX.X64_180000_db_home.zip

And now display the contents of the Oracle Home

In [48]:
display(Markdown('Directory Contents'))
containter_exec(db_container, f'ls -l {oracle_home}')

Directory Contents

total 300
drwxr-x--- 14 oracle oinstall  4096 Jun  4  2018 OPatch
drwxr-xr-x  2 oracle oinstall  4096 Feb  7  2018 QOpatch
drwxr-xr-x  5 oracle oinstall  4096 Feb  7  2018 R
drwxr-xr-x  2 oracle oinstall  4096 Jul 18  2018 addnode
drwxr-xr-x  6 oracle oinstall  4096 Feb  7  2018 apex
drwxr-xr-x  9 oracle oinstall  4096 Jul 18  2018 assistants
drwxr-xr-x  2 oracle oinstall  4096 Jul 18  2018 bin
drwxr-xr-x  4 oracle oinstall  4096 Jul 18  2018 clone
drwxr-xr-x  6 oracle oinstall  4096 Jul 18  2018 crs
drwxr-xr-x  3 oracle oinstall  4096 Feb  7  2018 css
drwxr-xr-x 11 oracle oinstall  4096 Feb  7  2018 ctx
drwxr-xr-x  7 oracle oinstall  4096 Jul 18  2018 cv
drwxr-xr-x  3 oracle oinstall  4096 Feb  7  2018 data
drwxr-xr-x  3 oracle oinstall  4096 Feb  7  2018 dbjava
drwxr-xr-x  2 oracle oinstall  4096 Feb  7  2018 dbs
drwxr-xr-x  5 oracle oinstall  4096 Jul 18  2018 deinstall
drwxr-xr-x  3 oracle oinstall  4096 Jul 18  2018 demo
drwxr-xr-x  3 oracle oinstall  4096 Feb  7  2018 diagnostics
drwxr-xr-x 13 oracle oinstall  4096 Feb  7  2018 dmu
drwxr-xr-x  4 oracle oinstall  4096 Jul 18  2018 drdaas
drwxr-xr-x  3 oracle oinstall  4096 Feb  7  2018 dv
-rw-r--r--  1 oracle oinstall   852 Aug 18  2015 env.ora
drwxr-xr-x  3 oracle oinstall  4096 Feb  7  2018 has
drwxr-xr-x  5 oracle oinstall  4096 Jul 18  2018 hs
drwxr-xr-x 10 oracle oinstall  4096 Jul 18  2018 install
drwxr-xr-x  2 oracle oinstall  4096 Feb  7  2018 instantclient
drwxr-x--- 13 oracle oinstall  4096 Jul 18  2018 inventory
drwxr-xr-x  8 oracle oinstall  4096 Jul 18  2018 javavm
drwxr-xr-x  3 oracle oinstall  4096 Feb  7  2018 jdbc
drwxr-xr-x  7 oracle oinstall  4096 Jul 18  2018 jdk
drwxr-xr-x  2 oracle oinstall  4096 Jul 18  2018 jlib
drwxr-xr-x 10 oracle oinstall  4096 Jul 18  2018 ldap
drwxr-xr-x  3 oracle oinstall 16384 Jul 18  2018 lib
drwxr-xr-x  2 oracle oinstall  4096 Feb  7  2018 log
drwxr-xr-x  9 oracle oinstall  4096 Jul 18  2018 md
drwxr-xr-x  4 oracle oinstall  4096 Feb  7  2018 mgw
drwxr-xr-x 10 oracle oinstall  4096 Jul 18  2018 network
drwxr-xr-x  5 oracle oinstall  4096 Jul 18  2018 nls
drwxr-xr-x  8 oracle oinstall  4096 Feb  7  2018 odbc
drwxr-xr-x  5 oracle oinstall  4096 Feb  7  2018 olap
drwxr-xr-x  7 oracle oinstall  4096 Jul 18  2018 opmn
drwxr-xr-x  4 oracle oinstall  4096 Feb  7  2018 oracore
drwxr-xr-x  7 oracle oinstall  4096 Jul 18  2018 ord
drwxr-xr-x  3 oracle oinstall  4096 Feb  7  2018 ordim
drwxr-xr-x  4 oracle oinstall  4096 Feb  7  2018 ords
drwxr-xr-x  3 oracle oinstall  4096 Feb  7  2018 oss
drwxr-xr-x  8 oracle oinstall  4096 Jul 18  2018 oui
drwxr-xr-x  4 oracle oinstall  4096 Feb  7  2018 owm
drwxr-xr-x  5 oracle oinstall  4096 Jul 18  2018 perl
drwxr-xr-x  6 oracle oinstall  4096 Feb  7  2018 plsql
drwxr-xr-x  6 oracle oinstall  4096 Jul 18  2018 precomp
drwxr-xr-x  4 oracle oinstall  4096 Feb  7  2018 racg
drwxr-xr-x 13 oracle oinstall  4096 Jul 18  2018 rdbms
drwxr-xr-x  3 oracle oinstall  4096 Feb  7  2018 relnotes
-rwx------  1 oracle oinstall   638 Jul 18  2018 root.sh
-rwx------  1 oracle oinstall   786 Feb  7  2018 root.sh.old
-rw-r-----  1 oracle oinstall    10 Feb  7  2018 root.sh.old.1
-rwxr-x---  1 oracle oinstall  1783 Mar  8  2017 runInstaller
-rw-r--r--  1 oracle oinstall  2927 Oct 14  2016 schagent.conf
drwxr-xr-x  5 oracle oinstall  4096 Feb  7  2018 sdk
drwxr-xr-x  3 oracle oinstall  4096 Feb  7  2018 slax
drwxr-xr-x 22 oracle oinstall  4096 Feb  7  2018 sqldeveloper
drwxr-xr-x  3 oracle oinstall  4096 Feb  7  2018 sqlj
drwxr-xr-x  5 oracle oinstall  4096 Jul 18  2018 sqlpatch
drwxr-xr-x  6 oracle oinstall  4096 Jul 18  2018 sqlplus
drwxr-xr-x  6 oracle oinstall  4096 Feb  7  2018 srvm
drwxr-xr-x  5 oracle oinstall  4096 Feb  7  2018 suptools
drwxr-xr-x  3 oracle oinstall  4096 Feb  7  2018 ucp
drwxr-xr-x  4 oracle oinstall  4096 Feb  7  2018 usm
drwxr-xr-x  2 oracle oinstall  4096 Feb  7  2018 utl
drwxr-xr-x  3 oracle oinstall  4096 Feb  7  2018 wwg
drwxr-x---  7 oracle oinstall  4096 Feb  7  2018 xdk

The next step is to create an Oracle Installer response file to reflect the paremeters we've defined. We're only going to perform a software only install.

In [49]:
script = f'''oracle.install.responseFileVersion=/oracle/install/rspfmt_dbinstall_response_schema_v18.0.0
oracle.install.option=INSTALL_DB_SWONLY
UNIX_GROUP_NAME=dba
INVENTORY_LOCATION={oracle_base}/oraInventory
ORACLE_BASE={oracle_base}
ORACLE_HOME={oracle_home}
oracle.install.db.InstallEdition=EE
oracle.install.db.OSDBA_GROUP=dba
oracle.install.db.OSBACKUPDBA_GROUP=dba
oracle.install.db.OSDGDBA_GROUP=dba
oracle.install.db.OSKMDBA_GROUP=dba
oracle.install.db.OSRACDBA_GROUP=dba
'''

copy_string_to_file(script, f'{oracle_home}/db_install.rsp', db_container)

Run the Oracle Installer

Now we can run the Oracle Installer in silent mode with a response file we've just created.

In [50]:
containter_exec(db_container, f'{oracle_home}/runInstaller -silent -force -waitforcompletion -responsefile {oracle_home}/db_install.rsp -ignorePrereqFailure')
Launching Oracle Database Setup Wizard...

[WARNING] [INS-32055] The Central Inventory is located in the Oracle base.
   ACTION: Oracle recommends placing this Central Inventory in a location outside the Oracle base directory.
[WARNING] [INS-13013] Target environment does not meet some mandatory requirements.
   CAUSE: Some of the mandatory prerequisites are not met. See logs for details. /tmp/InstallActions2019-03-19_02-40-31PM/installActions2019-03-19_02-40-31PM.log
   ACTION: Identify the list of failed prerequisite checks from the log: /tmp/InstallActions2019-03-19_02-40-31PM/installActions2019-03-19_02-40-31PM.log. Then either from the log file or from installation manual find the appropriate configuration to meet the prerequisites and fix it manually.
The response file for this session can be found at:
 /u01/app/oracle/product/18.0.0/dbhome_1/install/response/db_2019-03-19_02-40-31PM.rsp


You can find the log of this install session at:
 /tmp/InstallActions2019-03-19_02-40-31PM/installActions2019-03-19_02-40-31PM.log

As a root user, execute the following script(s):
1. /u01/app/oracle/oraInventory/orainstRoot.sh
	2. /u01/app/oracle/product/18.0.0/dbhome_1/root.sh

Execute /u01/app/oracle/oraInventory/orainstRoot.sh on the following nodes:
[oracle_db]

Execute /u01/app/oracle/product/18.0.0/dbhome_1/root.sh on the following nodes: 
[oracle_db]

Successfully Setup Software with warning(s).
Moved the install session logs to:

/u01/app/oracle/oraInventory/logs/InstallActions2019-03-19_02-40-31PM

In [51]:
containter_root_exec(db_container,f'/bin/bash {oracle_base}/oraInventory/orainstRoot.sh')
containter_root_exec(db_container,f'/bin/bash {oracle_home}/root.sh')
Changing permissions of /u01/app/oracle/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.
Changing groupname of /u01/app/oracle/oraInventory to dba.
The execution of the script is complete.
Check /u01/app/oracle/product/18.0.0/dbhome_1/install/root_oracle_db_2019-03-19_14-42-43-621215020.log for the output of root script

Commit the container to create an image

And finally we can commit the container creating an image for future use.

In [52]:
repository_name = 'dominicgiles'
db_container.commit(repository=repository_name,tag=f'db{oracle_version}')
Out[52]:
<Image: 'dominicgiles:db18.0.0'>

Tidy Up

The following code is included to enable you to quickly drop the container and potentially the immage.

In [53]:
db_container.stop()
db_container.remove()
In [54]:
list_containers(client)
Name Short id Status IP Address
In [55]:
list_images(client)
Tag id Size
dominicgiles:db18.0.0 cc75a47617 10.8 GB
linux_for_db18.0.0:latest b350079361 926.8 MB
oraclelinux:7-slim c3d8693881 117.3 MB
giaas.us.oracle.com:5001/dominic.giles/104213:db19000 808b7de5b5 7.8 GB
dbtools-docker.dockerhub-den.oraclecorp.com/oracle/sdw:18.4.0 af2d2f4838 1.8 GB
dbtools-docker.dockerhub-den.oraclecorp.com/oracle/database:18.3.0-ee c9eae81d87 8.4 GB
In [ ]:
#client.images.remove(image.id)

#list_images()
Comments

Jupyter, Python, Oracle and Docker Part 1

I'm releasing a few videos and blog entries describing my development and test environments. You can find them below

Part 1



Part 2




The first discusses what Jupyter Notebooks are and the second describes the installation of the environment I use.

There should be two or three more to follow.
Comments

Jupyter Notebooks, Python and Oracle Installation on Mac

OracleDockerSetupMac

Installing Jupyter-lab and Docker environment on Mac

The following walk-through guides you through the steps needed to set up your enviroment to run Jupyter-lab, Oracle and Docker to build and run docker images for testing. This should work for either an on premises install or on Oracle's cloud using IaaS (Compute). This walkthough will serve primarily as a reminder to myself.

Prerequisites

I tested this walkthrough on Mac with High Sierra but it should work on any moderately modern variant. I'm also assuming a few other things

  • You have admin access.
  • Comfortable using the Terminal utility.

Install

The install is pretty simple. It consists of Installing Homebrew, setting up python, installing Oracle Instant client, installing Git and then cloning this directory to the server. Lets start with setting up the Python Environment

Homebrew Install

Homebrew is equivalent on Mac to yum or apt-get on Linux. The beauty of it is is that it enables you to install packages in "user" space rather than "admin". It's easy to use and incredibly simple to install. You can either visit the website yourself and follow their instructions or within the terminal run the following command

/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

It will prompt you through the install. NOTE : It's possible that it may prompt you to install the XCode command line tools. Say yes.

Python Setup

Mac has Python installed but it's 2.7 and comes with a whole lot of packages that are needed for this exercise. Mucking around with it's version can lead to awhole lot of trouble. The good news is we can use brew to install an upto date variant. To do this just run the command

brew install python3

This might take a little while but it's worth doing. Brew should also install pip3 as part of the install but it's possible that this failed. To make sure it's usable simply run the following command

brew unlink python && brew link python

pip3 should now work and running a command like

pip3 list
Package    Version
---------- -------
pip        18.1 
setuptools 40.6.3
virtualenv 16.3.0
wheel      0.32.3

Will show you the installed modules.

Next we can create a virtual environment and enable it.

virtualenv -p /usr/local/bin/python3 myvirtualenv
source myvirtualenv/bin/activate

This will create a directory called myvirtualenv (you can call it what you like) with it's own version of the python interpreter and pip. Once we "active it", any library we install will only be in this directory and won't effect the system as a whole.

Docker Install

You have a couple of choices here. You could just run homebrew to install docker

brew install docker

Which works fine or you might want to use the semi graphical version "Docker Desktop for Mac". This has the small advantage of notifying you of available updates. If so you can download it here

https://hub.docker.com/editions/community/docker-ce-desktop-mac

Git Installation

We now need to install Git which is useful for managing and versioning code. That might not be a requirement for you but it also makes it very simple to clone existing repostories. Installing it is very simple.

brew install git

We can now clone my IPython/Jupyter notebooks from github which provide you with the code for creating your own Oracle Docker Images.

git clone https://github.com/domgiles/JuypterLabWork.git

This will create a directory call JuypterLabWork

Installing Oracle Instant Client

Installing the Oracle Instant client takes a few more steps on Mac than it does on Linux. Simply vist

https://www.oracle.com/technetwork/database/database-technologies/instant-client/downloads/index.html

Select Instant Client for Mac OS X (Intel x86) (32-bit and 64-bit) and on the next page select the Version 18.1.0.0.0 (64-bit) Basic Package. When it's downloaded simply unzip it into your home directory.

Installing Jupyter-Lab

In the JupyterLabWork directory that was created when we ran the git clone command there's a file called requirments.txt. This is a list of modules needed to run the notebooks in that directory. To install them all you need to do is to run the command

pip install -r requirements.txt

NOTE : You don't need to use pip3 when you have your virtual environment activated.

This will install all of the needed modules. From there all we need to do is to run the command

jupyter-lab

This should take you directly into a browser and the the jupyter environment.

jupter lab image

That's it.

Comments

Jupyter Notebooks, Python and Oracle Installation on Linux

OracleDockerSetupLinux

Installing Jupyter-lab and Docker environment on Linux

The following walk-through guides you through the steps needed to set up your enviroment to run Jupyter-lab, Oracle and Docker to build and run docker images for testing. This should work for either an on premises install or on Oracle's cloud using IaaS (Compute). This walkthough will serve primarily as a reminder to myself.

Prerequisites

I'm making the assumption that you're running on Linux (I've a similar walkthrough for mac). In my example I'm using Oracle Enterprise Linux 7 (OEL). I'm also assuming a few other things

  • Python 3.6 or higher is installed
  • You have access to root either directly or via sudo. In this example I'm installing everything in the Oracle user account that has sudo privelege
  • Docker is installed. If it isn't, see this excellent guide. If you aren't running as root also make sure you follow this final step as well.

Install

The install is pretty simple. It consists of setting up python, installing Oracle Instant client, installing Git and then cloning this directory to the server. Lets start with setting up the Python Environment

Python Setup

By default OEL 7 runs Python 2 rather than Python 3 which is likely to change in the future, until then we have a few steps we need to run through. The first is to install pip and virtualenv. We can easily do this on OEL with yum

sudo yum install python36-pip

If this command fails saying something like No package python36-pip available. then you'll need to edit the yum config file at /etc/yum.repos.d/public-yum-ol7.repo and enable the software development repos.

The next step is to install virtualenv. Virtualenv enables us to create isolated sandpits to develop Python applications without running into module or library conflicts. Once we have pip installed it's very simple to install

sudo pip3.6 install virtualenv

Next we can create a virtual environment and enable it.

virtualenv -p /usr/bin/python36 myvirtualenv
source myvirtualenv/bin/activate

This will create a directory called myvirtualenv (you can call it what you like) with it's own version of the python interpreter and pip. Once we "active it", any library we install will only be in this directory and won't effect the system as a whole. You should see you command prompt change when you activate it. It should look something like this

[04:59 PM : oracle@ora18server ~]$ virtualenv -p /usr/bin/python36 myvirtualenv
Running virtualenv with interpreter /usr/bin/python36
Using base prefix '/usr'
  No LICENSE.txt / LICENSE found in source
New python executable in /home/oracle/myvirtualenv/bin/python36
Also creating executable in /home/oracle/myvirtualenv/bin/python
Installing setuptools, pip, wheel...
done.
[05:00 PM : oracle@ora18server ~]$ source myvirtualenv/bin/activate
(myvirtualenv) [05:00 PM : oracle@ora18server ~]$ ls
Desktop       ora18server-certificate.crt  sql    swingbench
myvirtualenv  OracleUtils                  sqlcl  wallet
(myvirtualenv) [05:00 PM : oracle@ora18server ~]$

Running the following command will show what Python models we have installed at this point.

(myvirtualenv) [05:00 PM : oracle@ora18server ~]$ pip list
Package    Version
---------- -------
pip        19.0.3 
setuptools 40.8.0 
wheel      0.33.1

Which shouldn't be very many.

Git Installation

We now need to install Git which is useful for managing and versioning code. That might not be a requirement for you but it also makes it very simple to clone existing repostories. Installing it is very simple.

sudo yum install git

We can now clone my IPython/Jupyter notebooks from github which provides you with the code for creating your own Oracle Docker Images.

git clone https://github.com/domgiles/JuypterLabWork.git

This will create a directory call JuypterLabWork

Installing Oracle Instant Client

One of the recent updates to Oracles install models is the support for RPMs and yum installations without the need for click through agreements. This makes it very simple to install a client with a single command

sudo yum install oracle-instantclient18.3-basic

This will typically install the the software into /usr/lib/oracle/18.3

Installing Jupyter-Lab

In the JupyterLabWork directory that was created when we ran the git clone command there's a file called requirments.txt. This is a list of modules needed to run the notebooks in that directory. To install them all you need to do is to run the command

pip install -r requirements.txt

This will install all of the needed modules. From there all we need to do is to run the command

jupyter-lab

If you're running directly on a workstation or virtual machine and have a browser installed it should take you directly into the jupyter environment.

jupter lab image

If you're running headless then when jupyter-lab starts it should give you a url that you can connect to. Look for something like

[I 12:52:30.450 NotebookApp] The Jupyter Notebook is running at:
[I 12:52:30.450 NotebookApp] http://oracle18cserver:8888/?token=f71e677e202f5fffc3d20fe458ff973e616e0dc3b8eaf072
[I 12:52:30.450 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).

That's it.

Comments

Connecting to Oracle ADB from Python

Connecting to Oracle ADB