Modified the build system for Mac to optionally build and deploy the KLayout Python Module (*.whl and *.egg) for LW-*.dmg (#1078)

* WIP: add "pymod"

* To fix the issue reported by GitHub ticket No.1040

* WIP: To build "pymod"

* Updated the build system for Mac.

* Updated the build system for Mac.

* Modified the build system for Mac to optionally build and deploy the KLayout Python Module (*.whl and *.egg) for LW-*.dmg

macbuild/ReadMe.md

@@ -1,6 +1,6 @@
 Relevant KLayout version: 0.27.9<br>
 Author: Kazzz-S<br>
-Last modified: 2022-04-30<br>
+Last modified: 2022-05-08<br>
 
 # 1. Introduction
 This directory **`macbuild`** contains different files required for building KLayout (http://www.klayout.de/) version 0.27.9 or later for different 64-bit macOS, including:
@@ -110,7 +110,7 @@ $ [python] ./build4mac.py
                         :   Ana3: use Python 3.8 from Anaconda3                          | 
                         :   HB39: use Python 3.9 from Homebrew                           | 
                         : HBAuto: use the latest Python 3.x auto-detected from Homebrew  | 
-   [-j|--jump2pymod]    : jump into <pymod> build (developer's use only)                 | disabled
+   [-P|--buildPymod]    : build and deploy Pymod (*.whl and *.egg) for LW-*.dmg          | disabled
    [-n|--noqtbinding]   : don't create Qt bindings for ruby scripts                      | disabled
    [-u|--noqtuitools]   : don't include uitools in Qt binding                            | disabled
    [-m|--make <option>] : option passed to 'make'                                        | '--jobs=4'
@@ -162,7 +162,7 @@ $ [python] ./build4mac.py
                         :   Ana3: use Python 3.8 from Anaconda3                          | 
                         :   HB39: use Python 3.9 from Homebrew                           | 
                         : HBAuto: use the latest Python 3.x auto-detected from Homebrew  | 
-   [-j|--jump2pymod]    : jump into <pymod> build (developer's use only)                 | disabled
+   [-P|--buildPymod]    : build and deploy Pymod (*.whl and *.egg) for LW-*.dmg          | disabled
    [-n|--noqtbinding]   : don't create Qt bindings for ruby scripts                      | disabled
    [-u|--noqtuitools]   : don't include uitools in Qt binding                            | disabled
    [-m|--make <option>] : option passed to 'make'                                        | '--jobs=4'
@@ -235,7 +235,7 @@ $ ./build4mac.py -q qt6macports -r mp27 -p mp38
 2. Confirm successful build (it will take about one hour depending on your machine spec).
 3. Run **`build4mac.py`** again with the same options used in 1. PLUS "-Y" to deploy executables and libraries under **`klayout.app`** bundle.<br>
    The buddy command-line tools (strm*) will also be deployed under **klayout.app/Contents/Buddy/** in this step.<br>
-   The KLayout Python Module (\*.whl, \*.egg) will be deployed under **klayout.app/Contents/pymod-dist/**.
+   If you use `--buildPymod` option in Step-1 and Step-3, the KLayout Python Module (\*.whl, \*.egg) will be built and deployed under **klayout.app/Contents/pymod-dist/**.
 
 ```
 $ ./build4mac.py -q qt6macports -r mp27 -p mp38 -Y
@@ -262,7 +262,7 @@ $ ./build4mac.py -q qt6brew -r hb27 -p hb38
 2. Confirm successful build (it will take about one hour depending on your machine spec).
 3. Run **`build4mac.py`** again with the same options used in 1. PLUS "-Y" to deploy executables and libraries under **`klayout.app`** bundle.<br>
    The buddy command-line tools (strm*) will also be deployed under **klayout.app/Contents/Buddy/** in this step.<br>
-   The KLayout Python Module (\*.whl, \*.egg) will be deployed under **klayout.app/Contents/pymod-dist/**.
+   If you use `--buildPymod` option in Step-1 and Step-3, the KLayout Python Module (\*.whl, \*.egg) will be built and deployed under **klayout.app/Contents/pymod-dist/**.
 
 ```
 $ ./build4mac.py -q qt6brew -r hb27 -p hb38 -Y
@@ -315,7 +315,7 @@ $ ./build4mac.py -q qt5ana3 -r ana3 -p ana3
 2. Confirm successful build (it will take about one hour depending on your machine spec).
 3. Run **`build4mac.py`** again with the same options used in 1. PLUS "-Y" to deploy executables and libraries under **`klayout.app`** bundle.<br>
    The buddy command-line tools (strm*) will also be deployed under **klayout.app/Contents/Buddy/** in this step.<br>
-   The KLayout Python Module (\*.whl, \*.egg) will be deployed under **klayout.app/Contents/pymod-dist/**.
+   If you use `--buildPymod` option in Step-1 and Step-3, the KLayout Python Module (\*.whl, \*.egg) will be built and deployed under **klayout.app/Contents/pymod-dist/**.
 
 ```
 $ ./build4mac.py -q qt5ana3 -r ana3 -p ana3 -Y

macbuild/build4mac.py

@@ -74,7 +74,7 @@ def GenerateUsage(platform):
     usage += "                        :   Ana3: use Python 3.8 from Anaconda3                          | \n"
     usage += "                        :   HB39: use Python 3.9 from Homebrew                           | \n"
     usage += "                        : HBAuto: use the latest Python 3.x auto-detected from Homebrew  | \n"
-    usage += "   [-j|--jump2pymod]    : jump into <pymod> build (developer's use only)                 | disabled\n"
+    usage += "   [-P|--buildPymod]    : build and deploy Pymod (*.whl and *.egg) for LW-*.dmg          | disabled\n"
     usage += "   [-n|--noqtbinding]   : don't create Qt bindings for ruby scripts                      | disabled\n"
     usage += "   [-u|--noqtuitools]   : don't include uitools in Qt binding                            | disabled\n"
     usage += "   [-m|--make <option>] : option passed to 'make'                                        | '--jobs=4'\n"
@@ -181,7 +181,7 @@ def Get_Default_Config():
         ModuleRuby   = "nil"
         ModulePython = "nil"
 
-    Jump2Pymod    = False
+    BuildPymod    = False
     NonOSStdLang  = False
     NoQtBindings  = False
     NoQtUiTools   = False
@@ -202,7 +202,7 @@ def Get_Default_Config():
     config['ModuleQt']      = ModuleQt          # Qt module to be used
     config['ModuleRuby']    = ModuleRuby        # Ruby module to be used
     config['ModulePython']  = ModulePython      # Python module to be used
-    config['Jump2Pymod']    = Jump2Pymod        # True to jump into <pymod> build
+    config['BuildPymod']    = BuildPymod        # True to build and deploy "Pymod"
     config['NonOSStdLang']  = NonOSStdLang      # True if non-OS-standard language is chosen
     config['NoQtBindings']  = NoQtBindings      # True if not creating Qt bindings for Ruby scripts
     config['NoQtUiTools']   = NoQtUiTools       # True if not to include QtUiTools in Qt binding
@@ -242,7 +242,7 @@ def Parse_CLI_Args(config):
     ModuleQt      = config['ModuleQt']
     ModuleRuby    = config['ModuleRuby']
     ModulePython  = config['ModulePython']
-    Jump2Pymod    = config['Jump2Pymod']
+    BuildPymod    = config['BuildPymod']
     NonOSStdLang  = config['NonOSStdLang']
     NoQtBindings  = config['NoQtBindings']
     NoQtUiTools   = config['NoQtUiTools']
@@ -271,11 +271,11 @@ def Parse_CLI_Args(config):
                     dest='type_python',
                     help="Python type=['nil', 'Sys', 'MP38', 'HB38', 'Ana3', 'HB39', 'HBAuto']" )
 
-    p.add_option( '-j', '--jump2pymod',
+    p.add_option( '-P', '--buildPymod',
                     action='store_true',
-                    dest='jump_to_pymod',
+                    dest='build_pymod',
                     default=False,
-                    help="jump into <pymod> build (developer's use only)" )
+                    help="build and deploy <Pymod> (disabled)" )
 
     p.add_option( '-n', '--noqtbinding',
                     action='store_true',
@@ -331,7 +331,7 @@ def Parse_CLI_Args(config):
         p.set_defaults( type_qt        = "qt6brew",
                         type_ruby      = "hb27",
                         type_python    = "hb38",
-                        jump_to_pymod  = False,
+                        build_pymod    = False,
                         no_qt_binding  = False,
                         no_qt_uitools  = False,
                         make_option    = "--jobs=4",
@@ -345,7 +345,7 @@ def Parse_CLI_Args(config):
         p.set_defaults( type_qt        = "qt6brew",
                         type_ruby      = "sys",
                         type_python    = "sys",
-                        jump_to_pymod  = False,
+                        build_pymod    = False,
                         no_qt_binding  = False,
                         no_qt_uitools  = False,
                         make_option    = "--jobs=4",
@@ -500,7 +500,7 @@ def Parse_CLI_Args(config):
     ModuleSet = ( choiceQt65, choiceRuby, choicePython )
 
     # (E) Set other parameters
-    Jump2Pymod   = opt.jump_to_pymod
+    BuildPymod   = opt.build_pymod
     NoQtBindings = opt.no_qt_binding
     NoQtUiTools  = opt.no_qt_uitools
     MakeOptions  = opt.make_option
@@ -525,14 +525,21 @@ def Parse_CLI_Args(config):
     if not DeploymentF and not DeploymentP:
         target  = "%s %s %s" % (Platform, Release, Machine)
         modules = "Qt=%s, Ruby=%s, Python=%s" % (ModuleQt, ModuleRuby, ModulePython)
-        message = "### You are going to build KLayout\n    for  <%s>\n    with <%s>...\n"
+        if BuildPymod:
+            pymodbuild = "enabled"
+        else:
+            pymodbuild = "disabled"
+        message = "### You are going to build KLayout\n    for  <%s>\n    with <%s>\n    with Pymod <%s>...\n"
         print("")
-        print( message % (target, modules) )
+        print( message % (target, modules, pymodbuild) )
     else:
         message = "### You are going to make "
         if DeploymentP:
             PackagePrefix = "LW-"
-            message      += "a lightweight (LW-) package excluding Qt[6|5], Ruby, and Python..."
+            if not BuildPymod:
+                message += "a lightweight (LW-) package excluding Qt5, Ruby, and Python..."
+            else:
+                message += "a lightweight (LW-) package with Pymod excluding Qt5, Ruby, and Python..."
         elif DeploymentF:
             if (ModuleRuby in RubySys) and (ModulePython in PythonSys):
                 PackagePrefix = "ST-"
@@ -546,7 +553,7 @@ def Parse_CLI_Args(config):
         print( "" )
         print( message )
         print( "" )
-        if CheckComOnly and not Jump2Pymod:
+        if CheckComOnly:
             sys.exit(0)
 
     #-----------------------------------------------------
@@ -557,7 +564,7 @@ def Parse_CLI_Args(config):
     config['ModuleQt']      = ModuleQt
     config['ModuleRuby']    = ModuleRuby
     config['ModulePython']  = ModulePython
-    config['Jump2Pymod']    = Jump2Pymod
+    config['BuildPymod']    = BuildPymod
     config['NonOSStdLang']  = NonOSStdLang
     config['NoQtBindings']  = NoQtBindings
     config['NoQtUiTools']   = NoQtUiTools
@@ -569,7 +576,15 @@ def Parse_CLI_Args(config):
     config['PackagePrefix'] = PackagePrefix
     config['DeployVerbose'] = DeployVerbose
     config['ModuleSet']     = ModuleSet
-    return config
+
+    if CheckComOnly:
+        pp = pprint.PrettyPrinter( indent=4, width=140 )
+        parameters = Get_Build_Parameters(config)
+        Build_pymod(parameters)
+        pp.pprint(parameters)
+        sys.exit(0)
+    else:
+        return config
 
 #------------------------------------------------------------------------------
 ## To run the main Bash script "build.sh" with appropriate options
@@ -588,7 +603,7 @@ def Get_Build_Parameters(config):
     ModuleQt      = config['ModuleQt']
     ModuleRuby    = config['ModuleRuby']
     ModulePython  = config['ModulePython']
-    Jump2Pymod    = config['Jump2Pymod']
+    BuildPymod    = config['BuildPymod']
     ModuleSet     = config['ModuleSet']
     NoQtBindings  = config['NoQtBindings']
     NoQtUiTools   = config['NoQtUiTools']
@@ -656,6 +671,13 @@ def Get_Build_Parameters(config):
     # (G) options to `make` tool
     if not MakeOptions == "":
         parameters['make_options'] = MakeOptions
+        try:
+            jobopt, number = MakeOptions.split('=')  # like '--jobs=4' ?
+            pnum = int(number)
+        except Exception:
+            parameters['num_parallel'] = 4  # default
+        else:
+            parameters['num_parallel'] = pnum
 
     # (H) about Ruby
     if ModuleRuby != "nil":
@@ -681,12 +703,13 @@ def Get_Build_Parameters(config):
     parameters['project_dir'] = ProjectDir
 
     # (K) Extra parameters needed for <pymod>
-    #     <pymod> will be built for:
+    #     <pymod> will be built if:
+    #       BuildPymod   = True
     #       Platform     = [ 'Monterey', 'BigSur', 'Catalina' ]
     #       ModuleRuby   = [ 'Ruby27MacPorts', 'Ruby27Brew', 'RubyAnaconda3' ]
     #       ModulePython = [ 'Python38MacPorts', 'Python38Brew',
     #                        'PythonAnaconda3',  'PythonAutoBrew' ]
-    parameters['Jump2Pymod']   = Jump2Pymod
+    parameters['BuildPymod']   = BuildPymod
     parameters['Platform']     = Platform
     parameters['ModuleRuby']   = ModuleRuby
     parameters['ModulePython'] = ModulePython
@@ -713,15 +736,19 @@ def Get_Build_Parameters(config):
 #------------------------------------------------------------------------------
 def Build_pymod(parameters):
     #---------------------------------------------------------------------------
-    # [1] <pymod> will be built for:
+    # [1] <pymod> will be built if:
+    #       BuildPymod   = True
     #       Platform     = [ 'Monterey', 'BigSur', 'Catalina' ]
     #       ModuleRuby   = [ 'Ruby27MacPorts', 'Ruby27Brew', 'RubyAnaconda3' ]
     #       ModulePython = [ 'Python38MacPorts', 'Python38Brew',
     #                        'PythonAnaconda3',  'PythonAutoBrew' ]
     #---------------------------------------------------------------------------
+    BuildPymod   = parameters['BuildPymod']
     Platform     = parameters['Platform']
     ModuleRuby   = parameters['ModuleRuby']
     ModulePython = parameters['ModulePython']
+    if not BuildPymod:
+        return 0
     if not Platform in [ 'Monterey', 'BigSur', 'Catalina' ]:
         return 0
     elif not ModuleRuby in [ 'Ruby27MacPorts', 'Ruby27Brew', 'RubyAnaconda3' ]:
@@ -787,7 +814,7 @@ def Build_pymod(parameters):
     print( "  ", command4 )
     print( "" )
     if parameters['check_cmd_only']:
-        sys.exit(0)
+        return 0
 
     #-----------------------------------------------------
     # [5] Invoke the main Python scripts; takes time:-)
@@ -843,7 +870,7 @@ def Build_pymod(parameters):
 # @return 0 on success; non-zero (1), otherwise
 #------------------------------------------------------------------------------
 def Run_Build_Command(parameters):
-    jump2pymod = parameters['Jump2Pymod']
+    jump2pymod = False  # default=False; set True to jump into pymod-build for debugging
 
     if not jump2pymod:
         #-----------------------------------------------------
@@ -974,10 +1001,14 @@ def Run_Build_Command(parameters):
         print( "", file=sys.stderr )
 
     #------------------------------------------------------------------------
-    # [5] Build <pymod> for some predetermined environments
+    # [5] Build <pymod> for some predetermined environments on demand
     #------------------------------------------------------------------------
-    ret = Build_pymod(parameters)
+    BuildPymod = parameters['BuildPymod']
-    return ret
+    if BuildPymod:
+        ret = Build_pymod(parameters)
+        return ret
+    else:
+        return 0
 
 #------------------------------------------------------------------------------
 ## For making a bundle (klayout.app), deploy built binaries and libraries
@@ -1003,6 +1034,7 @@ def Deploy_Binaries_For_Bundle(config, parameters):
     ModuleRuby     = config['ModuleRuby']
     ModulePython   = config['ModulePython']
 
+    BuildPymod     = parameters['BuildPymod']
     ProjectDir     = parameters['project_dir']
     MacBinDir      = parameters['bin']
     MacBuildDir    = parameters['build']
@@ -1013,13 +1045,16 @@ def Deploy_Binaries_For_Bundle(config, parameters):
     AbsMacBuildDir = "%s/%s" % (ProjectDir, MacBuildDir)
     AbsMacBuildLog = "%s/%s" % (ProjectDir, MacBuildLog)
 
-    try:
+    if BuildPymod:
-        PymodDistDir = parameters['pymod_dist']
+        try:
-        pymodDistDir = PymodDistDir[ModulePython]   # [ 'dist-MP3', 'dist-HB3', 'dist-ana3' ]
+            PymodDistDir = parameters['pymod_dist']
-    except KeyError:
+            pymodDistDir = PymodDistDir[ModulePython]   # [ 'dist-MP3', 'dist-HB3', 'dist-ana3' ]
-        pymodDistDir = ""
+        except KeyError:
+            pymodDistDir = ""
+        else:
+            pass
     else:
-        pass
+        pymodDistDir = ""
 
     print("")
     print( "##### Started deploying libraries and executables for <klayout.app> #####" )
@@ -1100,7 +1135,7 @@ def Deploy_Binaries_For_Bundle(config, parameters):
     os.makedirs(targetDirF)
     os.makedirs(targetDirM)
     os.makedirs(targetDirB)
-    if not pymodDistDir == "":
+    if BuildPymod and not pymodDistDir == "":
         os.makedirs(targetDirP)
 
 
@@ -1275,8 +1310,8 @@ def Deploy_Binaries_For_Bundle(config, parameters):
         buddy = os.path.basename(item)
         os.chmod( targetDirB + "/" + buddy, 0o0755 )
 
-    # (C) the pymod
+    # (C) the Pymod
-    if not pymodDistDir == "":
+    if BuildPymod and not pymodDistDir == "":
         for item in glob.glob( pymodDistDir + "/*.whl" ):
             shutil.copy2( item,  targetDirP )
         for item in glob.glob( pymodDistDir + "/*.egg" ):

macbuild/nightlyBuild.py

@@ -88,6 +88,11 @@ def Get_Build_Options( targetDic ):
             buildOp["brewA"]   = [ '-q', 'Qt5Brew',     '-r', 'HB27', '-p', 'HBAuto' ]
         elif target == "brewAHW":
             buildOp["brewAHW"] = [ '-q', 'Qt5Brew',     '-r', 'sys',  '-p', 'HBAuto' ]
+
+    if WithPymod:
+        buildOp["ports"] = buildOp["ports"] + ['--buildPymod']
+        buildOp["brew"]  = buildOp["brew"]  + ['--buildPymod']
+        buildOp["ana3"]  = buildOp["ana3"]  + ['--buildPymod']
     return buildOp
 
 #------------------------------------------------------------------------------
@@ -167,6 +172,7 @@ def Parse_CommandLine_Arguments():
     global Usage     # usage
     global Target    # target list
     global Build     # operation flag
+    global WithPymod # operation flag
     global QATest    # operation flag
     global QACheck   # operation flag
     global MakeDMG   # operation flag
@@ -194,6 +200,7 @@ def Parse_CommandLine_Arguments():
     Usage += "   [--target <list>] : 0='std', 1='ports', 2='brew', 3='brewHW', 4='ana3', | '%s'\n" % targetopt
     Usage += "                       5='brewA', 6='brewAHW'                              | \n"
     Usage += "   [--build] : build and deploy                                            | disabled\n"
+    Usage += "   [--pymod] : build and deploy Pymod, too                                 | disabled\n"
     Usage += "   [--test]  : run the QA Test                                             | disabled\n"
     Usage += "   [--check] : check the QA Test results                                   | disabled\n"
     Usage += "   [--makedmg|--cleandmg <srlno>] : make or clean DMGs                     | disabled\n"
@@ -204,12 +211,12 @@ def Parse_CommandLine_Arguments():
     Usage += "          $ ln -s ./macbuild/nightlyBuild.py .                             | \n"
     Usage += "                                                                           | \n"
     Usage += "      Regular sequence for using this script:                              | \n"
-    Usage += "          (1) $ ./nightlyBuild.py  --build                                 | \n"
+    Usage += "          (1) $ ./nightlyBuild.py  --build  --pymod                        | \n"
     Usage += "          (2)   (confirm the build results)                                | \n"
     Usage += "          (3) $ ./nightlyBuild.py  --test                                  | \n"
     Usage += "          (4) $ ./nightlyBuild.py  --check (confirm the QA Test results)   | \n"
     Usage += "          (5) $ ./nightlyBuild.py  --makedmg  1                            | \n"
-    Usage += "          (6) $ ./nightlyBuild.py  --upload  '0.27.4'                      | \n"
+    Usage += "          (6) $ ./nightlyBuild.py  --upload  '0.27.9'                      | \n"
     Usage += "          (7) $ ./nightlyBuild.py  --cleandmg 1                            | \n"
     Usage += "---------------------------------------------------------------------------+----------------------\n"
 
@@ -224,6 +231,12 @@ def Parse_CommandLine_Arguments():
                     default=False,
                     help='build and deploy' )
 
+    p.add_option( '--pymod',
+                    action='store_true',
+                    dest='with_pymod',
+                    default=False,
+                    help='build and deploy Pymod, too ' )
+
     p.add_option( '--test',
                     action='store_true',
                     dest='qa_test',
@@ -256,6 +269,7 @@ def Parse_CommandLine_Arguments():
 
     p.set_defaults( targets    = "%s" % targetopt,
                     build      = False,
+                    with_pymod = False,
                     qa_test    = False,
                     qa_check   = False,
                     makedmg    = "",
@@ -280,12 +294,13 @@ def Parse_CommandLine_Arguments():
         if idx in range(0, 7):
             Target.append( targetDic[idx] )
 
-    Build    = opt.build
+    Build     = opt.build
-    QATest   = opt.qa_test
+    WithPymod = opt.with_pymod
-    QACheck  = opt.qa_check
+    QATest    = opt.qa_test
-    MakeDMG  = False
+    QACheck   = opt.qa_check
-    CleanDMG = False
+    MakeDMG   = False
-    Upload   = False
+    CleanDMG  = False
+    Upload    = False
 
     if not opt.makedmg == "":
         MakeDMG = True