Modified the build system for Mac to optionally build and deploy the KLayout Python Module (*.whl and *.egg) for LW-*.dmg (#1078)

* WIP: add "pymod"

* To fix the issue reported by GitHub ticket No.1040

* WIP: To build "pymod"

* Updated the build system for Mac.

* Updated the build system for Mac.

* Modified the build system for Mac to optionally build and deploy the KLayout Python Module (*.whl and *.egg) for LW-*.dmg
This commit is contained in:
Kazunari Sekigawa 2022-05-11 03:25:30 +09:00 committed by Matthias Koefferlein
parent 35767344bd
commit 926dc8f702
3 changed files with 98 additions and 48 deletions

View File

@ -1,6 +1,6 @@
Relevant KLayout version: 0.27.9<br>
Author: Kazzz-S<br>
Last modified: 2022-04-30<br>
Last modified: 2022-05-08<br>
# 1. Introduction
This directory **`macbuild`** contains different files required for building KLayout (http://www.klayout.de/) version 0.27.9 or later for different 64-bit macOS, including:
@ -110,7 +110,7 @@ $ [python] ./build4mac.py
: Ana3: use Python 3.8 from Anaconda3 |
: HB39: use Python 3.9 from Homebrew |
: HBAuto: use the latest Python 3.x auto-detected from Homebrew |
[-j|--jump2pymod] : jump into <pymod> build (developer's use only) | disabled
[-P|--buildPymod] : build and deploy Pymod (*.whl and *.egg) for LW-*.dmg | disabled
[-n|--noqtbinding] : don't create Qt bindings for ruby scripts | disabled
[-u|--noqtuitools] : don't include uitools in Qt binding | disabled
[-m|--make <option>] : option passed to 'make' | '--jobs=4'
@ -162,7 +162,7 @@ $ [python] ./build4mac.py
: Ana3: use Python 3.8 from Anaconda3 |
: HB39: use Python 3.9 from Homebrew |
: HBAuto: use the latest Python 3.x auto-detected from Homebrew |
[-j|--jump2pymod] : jump into <pymod> build (developer's use only) | disabled
[-P|--buildPymod] : build and deploy Pymod (*.whl and *.egg) for LW-*.dmg | disabled
[-n|--noqtbinding] : don't create Qt bindings for ruby scripts | disabled
[-u|--noqtuitools] : don't include uitools in Qt binding | disabled
[-m|--make <option>] : option passed to 'make' | '--jobs=4'
@ -235,7 +235,7 @@ $ ./build4mac.py -q qt6macports -r mp27 -p mp38
2. Confirm successful build (it will take about one hour depending on your machine spec).
3. Run **`build4mac.py`** again with the same options used in 1. PLUS "-Y" to deploy executables and libraries under **`klayout.app`** bundle.<br>
The buddy command-line tools (strm*) will also be deployed under **klayout.app/Contents/Buddy/** in this step.<br>
The KLayout Python Module (\*.whl, \*.egg) will be deployed under **klayout.app/Contents/pymod-dist/**.
If you use `--buildPymod` option in Step-1 and Step-3, the KLayout Python Module (\*.whl, \*.egg) will be built and deployed under **klayout.app/Contents/pymod-dist/**.
```
$ ./build4mac.py -q qt6macports -r mp27 -p mp38 -Y
@ -262,7 +262,7 @@ $ ./build4mac.py -q qt6brew -r hb27 -p hb38
2. Confirm successful build (it will take about one hour depending on your machine spec).
3. Run **`build4mac.py`** again with the same options used in 1. PLUS "-Y" to deploy executables and libraries under **`klayout.app`** bundle.<br>
The buddy command-line tools (strm*) will also be deployed under **klayout.app/Contents/Buddy/** in this step.<br>
The KLayout Python Module (\*.whl, \*.egg) will be deployed under **klayout.app/Contents/pymod-dist/**.
If you use `--buildPymod` option in Step-1 and Step-3, the KLayout Python Module (\*.whl, \*.egg) will be built and deployed under **klayout.app/Contents/pymod-dist/**.
```
$ ./build4mac.py -q qt6brew -r hb27 -p hb38 -Y
@ -315,7 +315,7 @@ $ ./build4mac.py -q qt5ana3 -r ana3 -p ana3
2. Confirm successful build (it will take about one hour depending on your machine spec).
3. Run **`build4mac.py`** again with the same options used in 1. PLUS "-Y" to deploy executables and libraries under **`klayout.app`** bundle.<br>
The buddy command-line tools (strm*) will also be deployed under **klayout.app/Contents/Buddy/** in this step.<br>
The KLayout Python Module (\*.whl, \*.egg) will be deployed under **klayout.app/Contents/pymod-dist/**.
If you use `--buildPymod` option in Step-1 and Step-3, the KLayout Python Module (\*.whl, \*.egg) will be built and deployed under **klayout.app/Contents/pymod-dist/**.
```
$ ./build4mac.py -q qt5ana3 -r ana3 -p ana3 -Y

View File

@ -74,7 +74,7 @@ def GenerateUsage(platform):
usage += " : Ana3: use Python 3.8 from Anaconda3 | \n"
usage += " : HB39: use Python 3.9 from Homebrew | \n"
usage += " : HBAuto: use the latest Python 3.x auto-detected from Homebrew | \n"
usage += " [-j|--jump2pymod] : jump into <pymod> build (developer's use only) | disabled\n"
usage += " [-P|--buildPymod] : build and deploy Pymod (*.whl and *.egg) for LW-*.dmg | disabled\n"
usage += " [-n|--noqtbinding] : don't create Qt bindings for ruby scripts | disabled\n"
usage += " [-u|--noqtuitools] : don't include uitools in Qt binding | disabled\n"
usage += " [-m|--make <option>] : option passed to 'make' | '--jobs=4'\n"
@ -181,7 +181,7 @@ def Get_Default_Config():
ModuleRuby = "nil"
ModulePython = "nil"
Jump2Pymod = False
BuildPymod = False
NonOSStdLang = False
NoQtBindings = False
NoQtUiTools = False
@ -202,7 +202,7 @@ def Get_Default_Config():
config['ModuleQt'] = ModuleQt # Qt module to be used
config['ModuleRuby'] = ModuleRuby # Ruby module to be used
config['ModulePython'] = ModulePython # Python module to be used
config['Jump2Pymod'] = Jump2Pymod # True to jump into <pymod> build
config['BuildPymod'] = BuildPymod # True to build and deploy "Pymod"
config['NonOSStdLang'] = NonOSStdLang # True if non-OS-standard language is chosen
config['NoQtBindings'] = NoQtBindings # True if not creating Qt bindings for Ruby scripts
config['NoQtUiTools'] = NoQtUiTools # True if not to include QtUiTools in Qt binding
@ -242,7 +242,7 @@ def Parse_CLI_Args(config):
ModuleQt = config['ModuleQt']
ModuleRuby = config['ModuleRuby']
ModulePython = config['ModulePython']
Jump2Pymod = config['Jump2Pymod']
BuildPymod = config['BuildPymod']
NonOSStdLang = config['NonOSStdLang']
NoQtBindings = config['NoQtBindings']
NoQtUiTools = config['NoQtUiTools']
@ -271,11 +271,11 @@ def Parse_CLI_Args(config):
dest='type_python',
help="Python type=['nil', 'Sys', 'MP38', 'HB38', 'Ana3', 'HB39', 'HBAuto']" )
p.add_option( '-j', '--jump2pymod',
p.add_option( '-P', '--buildPymod',
action='store_true',
dest='jump_to_pymod',
dest='build_pymod',
default=False,
help="jump into <pymod> build (developer's use only)" )
help="build and deploy <Pymod> (disabled)" )
p.add_option( '-n', '--noqtbinding',
action='store_true',
@ -331,7 +331,7 @@ def Parse_CLI_Args(config):
p.set_defaults( type_qt = "qt6brew",
type_ruby = "hb27",
type_python = "hb38",
jump_to_pymod = False,
build_pymod = False,
no_qt_binding = False,
no_qt_uitools = False,
make_option = "--jobs=4",
@ -345,7 +345,7 @@ def Parse_CLI_Args(config):
p.set_defaults( type_qt = "qt6brew",
type_ruby = "sys",
type_python = "sys",
jump_to_pymod = False,
build_pymod = False,
no_qt_binding = False,
no_qt_uitools = False,
make_option = "--jobs=4",
@ -500,7 +500,7 @@ def Parse_CLI_Args(config):
ModuleSet = ( choiceQt65, choiceRuby, choicePython )
# (E) Set other parameters
Jump2Pymod = opt.jump_to_pymod
BuildPymod = opt.build_pymod
NoQtBindings = opt.no_qt_binding
NoQtUiTools = opt.no_qt_uitools
MakeOptions = opt.make_option
@ -525,14 +525,21 @@ def Parse_CLI_Args(config):
if not DeploymentF and not DeploymentP:
target = "%s %s %s" % (Platform, Release, Machine)
modules = "Qt=%s, Ruby=%s, Python=%s" % (ModuleQt, ModuleRuby, ModulePython)
message = "### You are going to build KLayout\n for <%s>\n with <%s>...\n"
if BuildPymod:
pymodbuild = "enabled"
else:
pymodbuild = "disabled"
message = "### You are going to build KLayout\n for <%s>\n with <%s>\n with Pymod <%s>...\n"
print("")
print( message % (target, modules) )
print( message % (target, modules, pymodbuild) )
else:
message = "### You are going to make "
if DeploymentP:
PackagePrefix = "LW-"
message += "a lightweight (LW-) package excluding Qt[6|5], Ruby, and Python..."
if not BuildPymod:
message += "a lightweight (LW-) package excluding Qt5, Ruby, and Python..."
else:
message += "a lightweight (LW-) package with Pymod excluding Qt5, Ruby, and Python..."
elif DeploymentF:
if (ModuleRuby in RubySys) and (ModulePython in PythonSys):
PackagePrefix = "ST-"
@ -546,7 +553,7 @@ def Parse_CLI_Args(config):
print( "" )
print( message )
print( "" )
if CheckComOnly and not Jump2Pymod:
if CheckComOnly:
sys.exit(0)
#-----------------------------------------------------
@ -557,7 +564,7 @@ def Parse_CLI_Args(config):
config['ModuleQt'] = ModuleQt
config['ModuleRuby'] = ModuleRuby
config['ModulePython'] = ModulePython
config['Jump2Pymod'] = Jump2Pymod
config['BuildPymod'] = BuildPymod
config['NonOSStdLang'] = NonOSStdLang
config['NoQtBindings'] = NoQtBindings
config['NoQtUiTools'] = NoQtUiTools
@ -569,6 +576,14 @@ def Parse_CLI_Args(config):
config['PackagePrefix'] = PackagePrefix
config['DeployVerbose'] = DeployVerbose
config['ModuleSet'] = ModuleSet
if CheckComOnly:
pp = pprint.PrettyPrinter( indent=4, width=140 )
parameters = Get_Build_Parameters(config)
Build_pymod(parameters)
pp.pprint(parameters)
sys.exit(0)
else:
return config
#------------------------------------------------------------------------------
@ -588,7 +603,7 @@ def Get_Build_Parameters(config):
ModuleQt = config['ModuleQt']
ModuleRuby = config['ModuleRuby']
ModulePython = config['ModulePython']
Jump2Pymod = config['Jump2Pymod']
BuildPymod = config['BuildPymod']
ModuleSet = config['ModuleSet']
NoQtBindings = config['NoQtBindings']
NoQtUiTools = config['NoQtUiTools']
@ -656,6 +671,13 @@ def Get_Build_Parameters(config):
# (G) options to `make` tool
if not MakeOptions == "":
parameters['make_options'] = MakeOptions
try:
jobopt, number = MakeOptions.split('=') # like '--jobs=4' ?
pnum = int(number)
except Exception:
parameters['num_parallel'] = 4 # default
else:
parameters['num_parallel'] = pnum
# (H) about Ruby
if ModuleRuby != "nil":
@ -681,12 +703,13 @@ def Get_Build_Parameters(config):
parameters['project_dir'] = ProjectDir
# (K) Extra parameters needed for <pymod>
# <pymod> will be built for:
# <pymod> will be built if:
# BuildPymod = True
# Platform = [ 'Monterey', 'BigSur', 'Catalina' ]
# ModuleRuby = [ 'Ruby27MacPorts', 'Ruby27Brew', 'RubyAnaconda3' ]
# ModulePython = [ 'Python38MacPorts', 'Python38Brew',
# 'PythonAnaconda3', 'PythonAutoBrew' ]
parameters['Jump2Pymod'] = Jump2Pymod
parameters['BuildPymod'] = BuildPymod
parameters['Platform'] = Platform
parameters['ModuleRuby'] = ModuleRuby
parameters['ModulePython'] = ModulePython
@ -713,15 +736,19 @@ def Get_Build_Parameters(config):
#------------------------------------------------------------------------------
def Build_pymod(parameters):
#---------------------------------------------------------------------------
# [1] <pymod> will be built for:
# [1] <pymod> will be built if:
# BuildPymod = True
# Platform = [ 'Monterey', 'BigSur', 'Catalina' ]
# ModuleRuby = [ 'Ruby27MacPorts', 'Ruby27Brew', 'RubyAnaconda3' ]
# ModulePython = [ 'Python38MacPorts', 'Python38Brew',
# 'PythonAnaconda3', 'PythonAutoBrew' ]
#---------------------------------------------------------------------------
BuildPymod = parameters['BuildPymod']
Platform = parameters['Platform']
ModuleRuby = parameters['ModuleRuby']
ModulePython = parameters['ModulePython']
if not BuildPymod:
return 0
if not Platform in [ 'Monterey', 'BigSur', 'Catalina' ]:
return 0
elif not ModuleRuby in [ 'Ruby27MacPorts', 'Ruby27Brew', 'RubyAnaconda3' ]:
@ -787,7 +814,7 @@ def Build_pymod(parameters):
print( " ", command4 )
print( "" )
if parameters['check_cmd_only']:
sys.exit(0)
return 0
#-----------------------------------------------------
# [5] Invoke the main Python scripts; takes time:-)
@ -843,7 +870,7 @@ def Build_pymod(parameters):
# @return 0 on success; non-zero (1), otherwise
#------------------------------------------------------------------------------
def Run_Build_Command(parameters):
jump2pymod = parameters['Jump2Pymod']
jump2pymod = False # default=False; set True to jump into pymod-build for debugging
if not jump2pymod:
#-----------------------------------------------------
@ -974,10 +1001,14 @@ def Run_Build_Command(parameters):
print( "", file=sys.stderr )
#------------------------------------------------------------------------
# [5] Build <pymod> for some predetermined environments
# [5] Build <pymod> for some predetermined environments on demand
#------------------------------------------------------------------------
BuildPymod = parameters['BuildPymod']
if BuildPymod:
ret = Build_pymod(parameters)
return ret
else:
return 0
#------------------------------------------------------------------------------
## For making a bundle (klayout.app), deploy built binaries and libraries
@ -1003,6 +1034,7 @@ def Deploy_Binaries_For_Bundle(config, parameters):
ModuleRuby = config['ModuleRuby']
ModulePython = config['ModulePython']
BuildPymod = parameters['BuildPymod']
ProjectDir = parameters['project_dir']
MacBinDir = parameters['bin']
MacBuildDir = parameters['build']
@ -1013,6 +1045,7 @@ def Deploy_Binaries_For_Bundle(config, parameters):
AbsMacBuildDir = "%s/%s" % (ProjectDir, MacBuildDir)
AbsMacBuildLog = "%s/%s" % (ProjectDir, MacBuildLog)
if BuildPymod:
try:
PymodDistDir = parameters['pymod_dist']
pymodDistDir = PymodDistDir[ModulePython] # [ 'dist-MP3', 'dist-HB3', 'dist-ana3' ]
@ -1020,6 +1053,8 @@ def Deploy_Binaries_For_Bundle(config, parameters):
pymodDistDir = ""
else:
pass
else:
pymodDistDir = ""
print("")
print( "##### Started deploying libraries and executables for <klayout.app> #####" )
@ -1100,7 +1135,7 @@ def Deploy_Binaries_For_Bundle(config, parameters):
os.makedirs(targetDirF)
os.makedirs(targetDirM)
os.makedirs(targetDirB)
if not pymodDistDir == "":
if BuildPymod and not pymodDistDir == "":
os.makedirs(targetDirP)
@ -1275,8 +1310,8 @@ def Deploy_Binaries_For_Bundle(config, parameters):
buddy = os.path.basename(item)
os.chmod( targetDirB + "/" + buddy, 0o0755 )
# (C) the pymod
if not pymodDistDir == "":
# (C) the Pymod
if BuildPymod and not pymodDistDir == "":
for item in glob.glob( pymodDistDir + "/*.whl" ):
shutil.copy2( item, targetDirP )
for item in glob.glob( pymodDistDir + "/*.egg" ):

View File

@ -88,6 +88,11 @@ def Get_Build_Options( targetDic ):
buildOp["brewA"] = [ '-q', 'Qt5Brew', '-r', 'HB27', '-p', 'HBAuto' ]
elif target == "brewAHW":
buildOp["brewAHW"] = [ '-q', 'Qt5Brew', '-r', 'sys', '-p', 'HBAuto' ]
if WithPymod:
buildOp["ports"] = buildOp["ports"] + ['--buildPymod']
buildOp["brew"] = buildOp["brew"] + ['--buildPymod']
buildOp["ana3"] = buildOp["ana3"] + ['--buildPymod']
return buildOp
#------------------------------------------------------------------------------
@ -167,6 +172,7 @@ def Parse_CommandLine_Arguments():
global Usage # usage
global Target # target list
global Build # operation flag
global WithPymod # operation flag
global QATest # operation flag
global QACheck # operation flag
global MakeDMG # operation flag
@ -194,6 +200,7 @@ def Parse_CommandLine_Arguments():
Usage += " [--target <list>] : 0='std', 1='ports', 2='brew', 3='brewHW', 4='ana3', | '%s'\n" % targetopt
Usage += " 5='brewA', 6='brewAHW' | \n"
Usage += " [--build] : build and deploy | disabled\n"
Usage += " [--pymod] : build and deploy Pymod, too | disabled\n"
Usage += " [--test] : run the QA Test | disabled\n"
Usage += " [--check] : check the QA Test results | disabled\n"
Usage += " [--makedmg|--cleandmg <srlno>] : make or clean DMGs | disabled\n"
@ -204,12 +211,12 @@ def Parse_CommandLine_Arguments():
Usage += " $ ln -s ./macbuild/nightlyBuild.py . | \n"
Usage += " | \n"
Usage += " Regular sequence for using this script: | \n"
Usage += " (1) $ ./nightlyBuild.py --build | \n"
Usage += " (1) $ ./nightlyBuild.py --build --pymod | \n"
Usage += " (2) (confirm the build results) | \n"
Usage += " (3) $ ./nightlyBuild.py --test | \n"
Usage += " (4) $ ./nightlyBuild.py --check (confirm the QA Test results) | \n"
Usage += " (5) $ ./nightlyBuild.py --makedmg 1 | \n"
Usage += " (6) $ ./nightlyBuild.py --upload '0.27.4' | \n"
Usage += " (6) $ ./nightlyBuild.py --upload '0.27.9' | \n"
Usage += " (7) $ ./nightlyBuild.py --cleandmg 1 | \n"
Usage += "---------------------------------------------------------------------------+----------------------\n"
@ -224,6 +231,12 @@ def Parse_CommandLine_Arguments():
default=False,
help='build and deploy' )
p.add_option( '--pymod',
action='store_true',
dest='with_pymod',
default=False,
help='build and deploy Pymod, too ' )
p.add_option( '--test',
action='store_true',
dest='qa_test',
@ -256,6 +269,7 @@ def Parse_CommandLine_Arguments():
p.set_defaults( targets = "%s" % targetopt,
build = False,
with_pymod = False,
qa_test = False,
qa_check = False,
makedmg = "",
@ -281,6 +295,7 @@ def Parse_CommandLine_Arguments():
Target.append( targetDic[idx] )
Build = opt.build
WithPymod = opt.with_pymod
QATest = opt.qa_test
QACheck = opt.qa_check
MakeDMG = False