From 4277645b2a7c859b10e3da3425ebf7985ef60617 Mon Sep 17 00:00:00 2001 From: Vertana Date: Mon, 27 Mar 2023 12:37:00 -0700 Subject: [PATCH 01/35] Consolidated Install Scripts and Improve README Install scripts have been consolidated for every non-Windows OS. Python Requirements were consolidated. README improved to work locally and provide more information. --- README.md | 122 ++++++++++++++++++++++++----------------- gui.sh | 14 ++--- gui_macos.sh | 13 ----- macos_setup.sh | 38 ------------- requirements.txt | 8 ++- requirements_macos.txt | 32 ----------- setup.sh | 96 ++++++++++++++++++++++++++++++++ ubuntu_setup.sh | 12 ---- upgrade.sh | 2 +- upgrade_macos.sh | 16 ------ 10 files changed, 178 insertions(+), 175 deletions(-) delete mode 100755 gui_macos.sh delete mode 100755 macos_setup.sh delete mode 100644 requirements_macos.txt create mode 100755 setup.sh delete mode 100755 ubuntu_setup.sh delete mode 100755 upgrade_macos.sh diff --git a/README.md b/README.md index f3af693..92cfecd 100644 --- a/README.md +++ b/README.md @@ -6,21 +6,29 @@ If you run on Linux and would like to use the GUI, there is now a port of it as ### Table of Contents -- [Tutorials](https://github.com/bmaltais/kohya_ss#tutorials) -- [Required Dependencies](https://github.com/bmaltais/kohya_ss#required-dependencies) -- [Installation](https://github.com/bmaltais/kohya_ss#installation) - - [CUDNN 8.6](https://github.com/bmaltais/kohya_ss#optional-cudnn-86) -- [Upgrading](https://github.com/bmaltais/kohya_ss#upgrading) -- [Launching the GUI](https://github.com/bmaltais/kohya_ss#launching-the-gui) -- [Dreambooth](https://github.com/bmaltais/kohya_ss#dreambooth) -- [Finetune](https://github.com/bmaltais/kohya_ss#finetune) -- [Train Network](https://github.com/bmaltais/kohya_ss#train-network) -- [LoRA](https://github.com/bmaltais/kohya_ss#lora) -- [Troubleshooting](https://github.com/bmaltais/kohya_ss#troubleshooting) - - [Page File Limit](https://github.com/bmaltais/kohya_ss#page-file-limit) - - [No module called tkinter](https://github.com/bmaltais/kohya_ss#no-module-called-tkinter) - - [FileNotFoundError](https://github.com/bmaltais/kohya_ss#filenotfounderror) -- [Change History](https://github.com/bmaltais/kohya_ss#change-history) +- [Tutorials](#tutorials) +- [Required Dependencies](#required-dependencies) + - [Linux/macOS](#linux-and-macos-dependencies) +- [Installation](#installation) + - [Linux/macOS](#linux-and-macos) + - [Windows](#windows) + - [CUDNN 8.6](#optional--cudnn-86) +- [Upgrading](#upgrading) + - [Windows](#windows-upgrade) + - [Linux/macOS](#linux-and-macos-upgrade) +- [Launching the GUI](#starting-gui-service) + - [Windows](#launching-the-gui-on-windows) + - [Linux/macOS](#launching-the-gui-on-linux-and-macos) + - [Direct Launch via Python Script](#launching-the-gui-directly-using-kohyaguipy) +- [Dreambooth](#dreambooth) +- [Finetune](#finetune) +- [Train Network](#train-network) +- [LoRA](#lora) +- [Troubleshooting](#troubleshooting) + - [Page File Limit](#page-file-limit) + - [No module called tkinter](#no-module-called-tkinter) + - [FileNotFoundError](#filenotfounderror) +- [Change History](#change-history) ## Tutorials @@ -39,32 +47,27 @@ If you run on Linux and would like to use the GUI, there is now a port of it as - Install [Git](https://git-scm.com/download/win) - Install [Visual Studio 2015, 2017, 2019, and 2022 redistributable](https://aka.ms/vs/17/release/vc_redist.x64.exe) +### Linux and macOS dependencies + +These dependencies are taken care of via `setup.sh` in the installation section. No additional steps should be needed unless the scripts inform you otherwise. + ## Installation ### Runpod Follow the instructions found in this discussion: https://github.com/bmaltais/kohya_ss/discussions/379 -### MacOS +### Linux and macOS In the terminal, run ``` git clone https://github.com/bmaltais/kohya_ss.git cd kohya_ss -bash macos_setup.sh +# May need to chmod +x ./setup.sh if you're on a machine with stricter security. +./setup.sh ``` During the accelerate config screen after running the script answer "This machine", "None", "No" for the remaining questions. - -### Ubuntu -In the terminal, run - -``` -git clone https://github.com/bmaltais/kohya_ss.git -cd kohya_ss -bash ubuntu_setup.sh -``` - -then configure accelerate with the same answers as in the Windows instructions when prompted. +These are the same answers as the Windows install. ### Windows @@ -110,21 +113,13 @@ Run the following commands to install: python .\tools\cudann_1.8_install.py ``` -## Upgrading MacOS - -When a new release comes out, you can upgrade your repo with the following commands in the root directory: - -```bash -upgrade_macos.sh -``` - Once the commands have completed successfully you should be ready to use the new version. MacOS support is not tested and has been mostly taken from https://gist.github.com/jstayco/9f5733f05b9dc29de95c4056a023d645 -## Upgrading Windows +## Upgrading -When a new release comes out, you can upgrade your repo with the following commands in the root directory: - -```powershell +The following commands will work from the root directory of the project if you'd prefer to not run scripts. +These commands will work on any OS. +```bash git pull .\venv\Scripts\activate @@ -132,20 +127,40 @@ git pull pip install --use-pep517 --upgrade -r requirements.txt ``` +### Windows Upgrade +When a new release comes out, you can upgrade your repo with the following commands in the root directory: + +```powershell +./upgrade.ps1 +``` + +### Linux and macOS Upgrade +You can cd into the root directory and simply run + +```bash +./upgrade.sh +``` + Once the commands have completed successfully you should be ready to use the new version. -## Launching the GUI using gui.bat or gui.ps1 - -The script can be run with several optional command line arguments: +# Starting GUI Service +The following command line arguments can be passed to the scripts on any OS to configure the underlying service. +``` --listen: the IP address to listen on for connections to Gradio. ---username: a username for authentication. ---password: a password for authentication. ---server_port: the port to run the server listener on. ---inbrowser: opens the Gradio UI in a web browser. +--username: a username for authentication. +--password: a password for authentication. +--server_port: the port to run the server listener on. +--inbrowser: opens the Gradio UI in a web browser. --share: shares the Gradio UI. +``` -These command line arguments can be passed to the UI function as keyword arguments. To launch the Gradio UI, run the script in a terminal with the desired command line arguments, for example: +### Launching the GUI on Windows + +The two scripts to launch the GUI on Windows are gui.ps1 and gui.bat in the root directory. +You can use whichever script you prefer. + +To launch the Gradio UI, run the script in a terminal with the desired command line arguments, for example: `gui.ps1 --listen 127.0.0.1 --server_port 7860 --inbrowser --share` @@ -153,14 +168,19 @@ or `gui.bat --listen 127.0.0.1 --server_port 7860 --inbrowser --share` -## Launching the GUI using kohya_gui.py +## Launching the GUI on Linux and macOS -To run the GUI, simply use this command: +Run the launcher script with the desired command line arguments similar to Windows. +`gui.sh --listen 127.0.0.1 --server_port 7860 --inbrowser --share` + +## Launching the GUI directly using kohya_gui.py + +To run the GUI directly bypassing the wrapper scripts, simply use this command from the root project directory: ``` .\venv\Scripts\activate -python.exe .\kohya_gui.py +python .\kohya_gui.py ``` ## Dreambooth diff --git a/gui.sh b/gui.sh index e4eca6f..bd7078f 100755 --- a/gui.sh +++ b/gui.sh @@ -1,13 +1,9 @@ -#!/bin/bash +#!/usr/bin/env bash # Activate the virtual environment -source venv/bin/activate - -# Validate the requirements and store the exit code -python tools/validate_requirements.py -exit_code=$? - -# If the exit code is 0, run the kohya_gui.py script with the command-line arguments -if [ $exit_code -eq 0 ]; then +source ./venv/bin/activate +python -V +# If the requirements are validated, run the kohya_gui.py script with the command-line arguments +if python tools/validate_requirements.py; then python kohya_gui.py "$@" fi diff --git a/gui_macos.sh b/gui_macos.sh deleted file mode 100755 index 4a0bfb8..0000000 --- a/gui_macos.sh +++ /dev/null @@ -1,13 +0,0 @@ -#!/bin/bash - -# Activate the virtual environment -source venv/bin/activate - -# Validate the requirements and store the exit code -python tools/validate_requirements.py --requirements requirements_macos.txt -exit_code=$? - -# If the exit code is 0, run the kohya_gui.py script with the command-line arguments -if [ $exit_code -eq 0 ]; then - python kohya_gui.py "$@" -fi diff --git a/macos_setup.sh b/macos_setup.sh deleted file mode 100755 index 4de8417..0000000 --- a/macos_setup.sh +++ /dev/null @@ -1,38 +0,0 @@ -#!/bin/bash -# The initial setup script to prep the environment on macOS -# xformers has been omitted as that is for Nvidia GPUs only - -if ! command -v brew >/dev/null; then - echo "Please install homebrew first. This is a requirement for the remaining setup." - echo "You can find that here: https://brew.sh" - exit 1 -fi - -# Install base python packages -echo "Installing Python 3.10 if not found." -brew ls --versions python@3.10 >/dev/null || brew install python@3.10 -echo "Installing Python-TK 3.10 if not found." -brew ls --versions python-tk@3.10 >/dev/null || brew install python-tk@3.10 - -if command -v python3.10 >/dev/null; then - python3.10 -m venv venv - source venv/bin/activate - - # DEBUG ONLY - #pip install pydevd-pycharm~=223.8836.43 - - # Tensorflow installation - if wget https://github.com/apple/tensorflow_macos/releases/download/v0.1alpha3/tensorflow_macos-0.1a3-cp38-cp38-macosx_11_0_arm64.whl /tmp; then - python -m pip install tensorflow==0.1a3 -f https://github.com/apple/tensorflow_macos/releases/download/v0.1alpha3/tensorflow_macos-0.1a3-cp38-cp38-macosx_11_0_arm64.whl - rm -f /tmp/tensorflow_macos-0.1a3-cp38-cp38-macosx_11_0_arm64.whl - fi - - pip install torch==2.0.0 torchvision==0.15.1 -f https://download.pytorch.org/whl/cpu/torch_stable.html - python -m pip install --use-pep517 --upgrade -r requirements_macos.txt - accelerate config - echo -e "Setup finished! Run ./gui_macos.sh to start." -else - echo "Python not found. Please ensure you install Python." - echo "The brew command for Python 3.10 is: brew install python@3.10" - exit 1 -fi \ No newline at end of file diff --git a/requirements.txt b/requirements.txt index 8e47439..4ee4eec 100644 --- a/requirements.txt +++ b/requirements.txt @@ -7,7 +7,8 @@ diffusers[torch]==0.10.2 easygui==0.98.3 einops==0.6.0 ftfy==6.1.1 -gradio==3.19.1 +gradio==3.19.1; sys_platform != 'darwin' +gradio==3.23.0; sys_platform == 'darwin' lion-pytorch==0.0.6 opencv-python==4.7.0.68 pytorch-lightning==1.9.0 @@ -22,8 +23,9 @@ fairscale==0.4.13 requests==2.28.2 timm==0.6.12 # tensorflow<2.11 -huggingface-hub==0.12.0 -tensorflow==2.10.1 +huggingface-hub==0.12.0; sys_platform != 'darwin' +huggingface-hub==0.13.0; sys_platform == 'darwin' +tensorflow==2.10.1; sys_platform != 'darwin' # For locon support lycoris_lora==0.1.2 # for kohya_ss library diff --git a/requirements_macos.txt b/requirements_macos.txt deleted file mode 100644 index 4ee4eec..0000000 --- a/requirements_macos.txt +++ /dev/null @@ -1,32 +0,0 @@ -accelerate==0.15.0 -albumentations==1.3.0 -altair==4.2.2 -bitsandbytes==0.35.0 -dadaptation==1.5 -diffusers[torch]==0.10.2 -easygui==0.98.3 -einops==0.6.0 -ftfy==6.1.1 -gradio==3.19.1; sys_platform != 'darwin' -gradio==3.23.0; sys_platform == 'darwin' -lion-pytorch==0.0.6 -opencv-python==4.7.0.68 -pytorch-lightning==1.9.0 -safetensors==0.2.6 -tensorboard==2.10.1 -tk==0.1.0 -toml==0.10.2 -transformers==4.26.0 -voluptuous==0.13.1 -# for BLIP captioning -fairscale==0.4.13 -requests==2.28.2 -timm==0.6.12 -# tensorflow<2.11 -huggingface-hub==0.12.0; sys_platform != 'darwin' -huggingface-hub==0.13.0; sys_platform == 'darwin' -tensorflow==2.10.1; sys_platform != 'darwin' -# For locon support -lycoris_lora==0.1.2 -# for kohya_ss library -. \ No newline at end of file diff --git a/setup.sh b/setup.sh new file mode 100755 index 0000000..6cba233 --- /dev/null +++ b/setup.sh @@ -0,0 +1,96 @@ +#!/usr/bin/env bash + +if [[ "$OSTYPE" == "linux-gnu"* ]]; then + # Check if root or sudo + root=true + if [ "$EUID" -ne 0 ]; then + root=false + fi + + distro="$(python -mplatform)" + if "$distro" | grep -qi "Ubuntu"; then + echo "Ubuntu detected." + echo "Installing Python TK if not found on the system." + if [ ! $(dpkg-query -W -f='${Status}' python3-tk 2>/dev/null | grep -c "ok installed") -eq 0 ]; then + if [ root = true ]; then + apt-get install python3-tk + else + echo "This script needs to be run as root or via sudo to install packages." + exit 1 + fi + else + echo "Python TK found! Skipping install!" + fi + elif "$distro" | grep -Eqi "Fedora|CentOS|Redhat"; then + if ! rpm -qa | grep -qi python3-tkinter; then + if [ root = true ]; then + dnf install python3-tkinter + else + echo "This script needs to be run as root or via sudo to install packages." + exit 1 + fi + fi + fi + + python3 -m venv venv + source venv/bin/activate + pip install torch==1.12.1+cu116 torchvision==0.13.1+cu116 --extra-index-url https://download.pytorch.org/whl/cu116 + pip install --use-pep517 --upgrade -r requirements.txt + pip install -U -I --no-deps https://github.com/C43H66N12O12S2/stable-diffusion-webui/releases/download/linux/xformers-0.0.14.dev0-cp310-cp310-linux_x86_64.whl + accelerate config + + echo -e "Setup finished! Run \e[0;92m./gui.sh\e[0m to start." +elif [[ "$OSTYPE" == "darwin"* ]]; then + # The initial setup script to prep the environment on macOS + # xformers has been omitted as that is for Nvidia GPUs only + + if ! command -v brew >/dev/null; then + echo "Please install homebrew first. This is a requirement for the remaining setup." + echo "You can find that here: https://brew.sh" + exit 1 + fi + + # Install base python packages + echo "Installing Python 3.10 if not found." + if ! brew ls --versions python@3.10 >/dev/null; then + brew install python@3.10 + else + echo "Python 3.10 found!" + fi + echo "Installing Python-TK 3.10 if not found." + if ! brew ls --versions python-tk@3.10 >/dev/null; then + brew install python-tk@3.10 + else + echo "Python Tkinter 3.10 found!" + fi + + if command -v python3.10 >/dev/null; then + python3.10 -m venv venv + source venv/bin/activate + + # DEBUG ONLY + #pip install pydevd-pycharm~=223.8836.43 + + # Tensorflow installation + if wget https://github.com/apple/tensorflow_macos/releases/download/v0.1alpha3/tensorflow_macos-0.1a3-cp38-cp38-macosx_11_0_arm64.whl /tmp; then + python -m pip install tensorflow==0.1a3 -f https://github.com/apple/tensorflow_macos/releases/download/v0.1alpha3/tensorflow_macos-0.1a3-cp38-cp38-macosx_11_0_arm64.whl + rm -f /tmp/tensorflow_macos-0.1a3-cp38-cp38-macosx_11_0_arm64.whl + fi + + pip install torch==2.0.0 torchvision==0.15.1 -f https://download.pytorch.org/whl/cpu/torch_stable.html + python -m pip install --use-pep517 --upgrade -r requirements.txt + accelerate config + echo -e "Setup finished! Run ./gui.sh to start." + else + echo "Python not found. Please ensure you install Python." + echo "The brew command for Python 3.10 is: brew install python@3.10" + exit 1 + fi +elif [[ "$OSTYPE" == "cygwin" ]]; then + # Cygwin is a standalone suite of Linux utilies on Windows + echo "This hasn't been validated on cygwin yet." +elif [[ "$OSTYPE" == "msys" ]]; then + # MinGW has the msys environment which is a standalone suite of Linux utilies on Windows + # "git bash" on Windows may also be detected as msys. + echo "This hasn't been validated in msys (mingw) on Windows yet." +fi diff --git a/ubuntu_setup.sh b/ubuntu_setup.sh deleted file mode 100755 index 1431155..0000000 --- a/ubuntu_setup.sh +++ /dev/null @@ -1,12 +0,0 @@ -#!/bin/bash -echo installing tk -sudo apt install python3-tk -python3 -m venv venv -source venv/bin/activate -pip install torch==1.12.1+cu116 torchvision==0.13.1+cu116 --extra-index-url https://download.pytorch.org/whl/cu116 -pip install --use-pep517 --upgrade -r requirements.txt -pip install -U -I --no-deps https://github.com/C43H66N12O12S2/stable-diffusion-webui/releases/download/linux/xformers-0.0.14.dev0-cp310-cp310-linux_x86_64.whl - -accelerate config - -echo -e "setup finished! run \e[0;92m./gui.sh\e[0m to start" diff --git a/upgrade.sh b/upgrade.sh index f01e7b7..8ed545f 100755 --- a/upgrade.sh +++ b/upgrade.sh @@ -1,4 +1,4 @@ -#!/bin/bash +#!/usr/bin/env bash # Check if there are any changes that need to be committed if [[ -n $(git status --short) ]]; then diff --git a/upgrade_macos.sh b/upgrade_macos.sh deleted file mode 100755 index 2e26c55..0000000 --- a/upgrade_macos.sh +++ /dev/null @@ -1,16 +0,0 @@ -#!/bin/bash - -# Check if there are any changes that need to be committed -if [[ -n $(git status --short) ]]; then - echo "There are changes that need to be committed. Please stash or undo your changes before running this script." >&2 - exit 1 -fi - -# Pull the latest changes from the remote repository -git pull - -# Activate the virtual environment -source venv/bin/activate - -# Upgrade the required packages -pip install --upgrade -r requirements_macos.txt From 8168e36326b11d1ce40c0009ed05ad69c048d013 Mon Sep 17 00:00:00 2001 From: jstayco <127801635+jstayco@users.noreply.github.com> Date: Mon, 27 Mar 2023 18:58:03 -0700 Subject: [PATCH 02/35] Update setup.sh Small comment added to clarify the purpose of the script for casual viewers. --- setup.sh | 2 ++ 1 file changed, 2 insertions(+) diff --git a/setup.sh b/setup.sh index 6cba233..87985ad 100755 --- a/setup.sh +++ b/setup.sh @@ -1,5 +1,7 @@ #!/usr/bin/env bash +# This file will be the host environment setup file for all operating systems other than base Windows. + if [[ "$OSTYPE" == "linux-gnu"* ]]; then # Check if root or sudo root=true From c79181b7ea06732575b2f529387573eacdfcb8ea Mon Sep 17 00:00:00 2001 From: jstayco <127801635+jstayco@users.noreply.github.com> Date: Mon, 27 Mar 2023 18:58:30 -0700 Subject: [PATCH 03/35] Update gui.sh Removed an unnecessary debug line. --- gui.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/gui.sh b/gui.sh index bd7078f..4fa2e35 100755 --- a/gui.sh +++ b/gui.sh @@ -2,7 +2,7 @@ # Activate the virtual environment source ./venv/bin/activate -python -V + # If the requirements are validated, run the kohya_gui.py script with the command-line arguments if python tools/validate_requirements.py; then python kohya_gui.py "$@" From 8bf3ecfd1f3725cfa442bd4ce720f81dd378ef4d Mon Sep 17 00:00:00 2001 From: JSTayco Date: Tue, 28 Mar 2023 15:52:16 -0700 Subject: [PATCH 04/35] Update setup.sh Linux distribution detection is much more robust. We also now include the Linux distribution family. That should help downstream Linux distributions as well. For example, Manjaro will now be detected as arch and Mint Linux will be detected as Ubuntu. --- setup.sh | 80 ++++++++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 78 insertions(+), 2 deletions(-) diff --git a/setup.sh b/setup.sh index 87985ad..60a3c08 100755 --- a/setup.sh +++ b/setup.sh @@ -9,8 +9,56 @@ if [[ "$OSTYPE" == "linux-gnu"* ]]; then root=false fi - distro="$(python -mplatform)" - if "$distro" | grep -qi "Ubuntu"; then + get_distro_name() { + local line + if [ -f /etc/os-release ]; then + # We search for the line starting with ID= + # Then we remove the ID= prefix to get the name itself + line="$(grep -Ei '^ID=' /etc/os-release)" + line=${line##*=} + echo "$line" + return 0 + elif command -v python >/dev/null; then + line="$(python -mplatform)" + echo "$line" + return 0 + elif command -v python3 >/dev/null; then + line="$(python3 -mplatform)" + echo "$line" + return 0 + else + line="None" + echo "$line" + return 1 + fi + } + + get_distro_family() { + local line + if [ -f /etc/os-release ]; then + # We search for the line starting with ID_LIKE= + # Then we remove the ID_LIKE= prefix to get the name itself + # This is the "type" of distro. For example, Ubuntu returns "debian". + if grep -Eiq '^ID_LIKE=' /etc/os-release >/dev/null; then + line="$(grep -Ei '^ID_LIKE=' /etc/os-release)" + line=${line##*=} + echo "$line" + return 0 + else + line="None" + echo "$line" + return 1 + fi + else + line="None" + echo "$line" + return 1 + fi + } + + distro=get_distro_name + family=get_distro_family + if "$distro" | grep -qi "Ubuntu" || "$family" | grep -qi "Ubuntu"; then echo "Ubuntu detected." echo "Installing Python TK if not found on the system." if [ ! $(dpkg-query -W -f='${Status}' python3-tk 2>/dev/null | grep -c "ok installed") -eq 0 ]; then @@ -32,6 +80,32 @@ if [[ "$OSTYPE" == "linux-gnu"* ]]; then exit 1 fi fi + elif "$distro" | grep -Eqi "arch" || "$family" | grep -qi "arch"; then + if ! pacman -Qi tk >/dev/null; then + if [ root = true ]; then + pacman -S tk + else + echo "This script needs to be run as root or via sudo to install packages." + exit 1 + fi + fi + elif "$distro" | grep -Eqi "opensuse" || "$family" | grep -qi "opensuse"; then + if ! rpm -qa | grep -qi python-tk; then + if [ root = true ]; then + zypper install python-tk + else + echo "This script needs to be run as root or via sudo to install packages." + exit 1 + fi + fi + elif [ "$distro" = "None" ] || [ "$family" = "None" ]; then + if [ "$distro" = "None" ]; then + echo "We could not detect your distribution of Linux. Please file a bug report on github with the contents of your /etc/os-release file." + fi + + if [ "$family" = "None" ]; then + echo "We could not detect the family of your Linux distribution. Please file a bug report on github with the contents of your /etc/os-release file." + fi fi python3 -m venv venv @@ -49,6 +123,8 @@ elif [[ "$OSTYPE" == "darwin"* ]]; then if ! command -v brew >/dev/null; then echo "Please install homebrew first. This is a requirement for the remaining setup." echo "You can find that here: https://brew.sh" + #shellcheck disable=SC2016 + echo 'The "brew" command should be in $PATH to be detected.' exit 1 fi From 51e843fe2ed68edbe38ef9cb49faaae61ab70578 Mon Sep 17 00:00:00 2001 From: JSTayco Date: Tue, 28 Mar 2023 16:02:20 -0700 Subject: [PATCH 05/35] Update setup.sh Fixed install commands to not require user input. --- setup.sh | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/setup.sh b/setup.sh index 60a3c08..9a2602e 100755 --- a/setup.sh +++ b/setup.sh @@ -63,7 +63,7 @@ if [[ "$OSTYPE" == "linux-gnu"* ]]; then echo "Installing Python TK if not found on the system." if [ ! $(dpkg-query -W -f='${Status}' python3-tk 2>/dev/null | grep -c "ok installed") -eq 0 ]; then if [ root = true ]; then - apt-get install python3-tk + apt install -y python3-tk else echo "This script needs to be run as root or via sudo to install packages." exit 1 @@ -74,7 +74,7 @@ if [[ "$OSTYPE" == "linux-gnu"* ]]; then elif "$distro" | grep -Eqi "Fedora|CentOS|Redhat"; then if ! rpm -qa | grep -qi python3-tkinter; then if [ root = true ]; then - dnf install python3-tkinter + dnf install python3-tkinter -y else echo "This script needs to be run as root or via sudo to install packages." exit 1 @@ -83,7 +83,7 @@ if [[ "$OSTYPE" == "linux-gnu"* ]]; then elif "$distro" | grep -Eqi "arch" || "$family" | grep -qi "arch"; then if ! pacman -Qi tk >/dev/null; then if [ root = true ]; then - pacman -S tk + pacman --noconfirm -S tk else echo "This script needs to be run as root or via sudo to install packages." exit 1 @@ -92,7 +92,7 @@ if [[ "$OSTYPE" == "linux-gnu"* ]]; then elif "$distro" | grep -Eqi "opensuse" || "$family" | grep -qi "opensuse"; then if ! rpm -qa | grep -qi python-tk; then if [ root = true ]; then - zypper install python-tk + zypper install -y python-tk else echo "This script needs to be run as root or via sudo to install packages." exit 1 From 307c433254af05735e70d0db174b6a31523878e2 Mon Sep 17 00:00:00 2001 From: JSTayco Date: Tue, 28 Mar 2023 18:07:18 -0700 Subject: [PATCH 06/35] The runpod update! Script now accepts long and short arguments. Script should now help with a runpod environment and ensure apt cache is updated before package install attempt. --- README.md | 2 ++ setup.sh | 95 ++++++++++++++++++++++++++++++++++++++++++++++++++++++- 2 files changed, 96 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 92cfecd..0a866b2 100644 --- a/README.md +++ b/README.md @@ -63,6 +63,8 @@ In the terminal, run git clone https://github.com/bmaltais/kohya_ss.git cd kohya_ss # May need to chmod +x ./setup.sh if you're on a machine with stricter security. +# There are additional options if needed for a runpod environment. +# Call 'setup.sh -h' or 'setup.sh --help' for more information. ./setup.sh ``` diff --git a/setup.sh b/setup.sh index 9a2602e..0fa1e4c 100755 --- a/setup.sh +++ b/setup.sh @@ -2,6 +2,51 @@ # This file will be the host environment setup file for all operating systems other than base Windows. +display_help() { + cat </dev/null | grep -c "ok installed") -eq 0 ]; then if [ root = true ]; then - apt install -y python3-tk + apt update -y && apt install -y python3-tk else echo "This script needs to be run as root or via sudo to install packages." exit 1 @@ -113,6 +183,29 @@ if [[ "$OSTYPE" == "linux-gnu"* ]]; then pip install torch==1.12.1+cu116 torchvision==0.13.1+cu116 --extra-index-url https://download.pytorch.org/whl/cu116 pip install --use-pep517 --upgrade -r requirements.txt pip install -U -I --no-deps https://github.com/C43H66N12O12S2/stable-diffusion-webui/releases/download/linux/xformers-0.0.14.dev0-cp310-cp310-linux_x86_64.whl + + # We need this extra package and setup if we are running in a runpod + if env_var_exists RUNPOD_POD_ID || env_var_exists RUNPOD_API_KEY; then + pip install tensorrt + ln -s "$VENV_DIR/lib/python3.10/site-packages/tensorrt/libnvinfer_plugin.so.8" \ + "$VENV_DIR/lib/python3.10/site-packages/tensorrt/libnvinfer_plugin.so.7" + ln -s "$VENV_DIR/lib/python3.10/site-packages/tensorrt/libnvinfer.so.8" \ + "$VENV_DIR/lib/python3.10/site-packages/tensorrt/libnvinfer.so.7" + ln -s "$VENV_DIR/lib/python3.10/site-packages/nvidia/cuda_runtime/lib/libcudart.so.12" \ + "$VENV_DIR/lib/python3.10/site-packages/nvidia/cuda_runtime/lib/libcudart.so.11.0" + + export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$VENV_DIR/lib/python3.10/site-packages/tensorrt/" + export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$VENV_DIR/lib/python3.10/site-packages/nvidia/cuda_runtime/lib/" + + # This is a non-interactive environment, so just directly call gui.sh after all setup steps are complete. + if command -v bash >/dev/null; then + bash "$DIR"/gui.sh + else + # This shouldn't happen, but we're going to try to help. + sh "$DIR"/gui.sh + fi + fi + accelerate config echo -e "Setup finished! Run \e[0;92m./gui.sh\e[0m to start." From 0b5a418b39586375c1380315bbda26d7834a2750 Mon Sep 17 00:00:00 2001 From: JSTayco Date: Wed, 29 Mar 2023 00:47:03 -0700 Subject: [PATCH 07/35] More options, better var handlng. Changed runpod detection to a variable to simplify maintenance and provide a mechanism for the user to force a runpod installation. Also, updated help message to acount for the change. --- setup.sh | 28 ++++++++++++++++++---------- 1 file changed, 18 insertions(+), 10 deletions(-) diff --git a/setup.sh b/setup.sh index 0fa1e4c..9af9320 100755 --- a/setup.sh +++ b/setup.sh @@ -10,22 +10,25 @@ The following options are useful in a runpod environment, but will not affect a local machine install. Usage: - setup.sh -b dev -d /workspace/kohya_ss - setup.sh --branch=dev --dir=/workspace/kohya_ss + setup.sh -b dev -d /workspace/kohya_ss -g https://mycustom.repo.tld/custom_fork.git + setup.sh --branch=dev --dir=/workspace/kohya_ss --git-repo=https://mycustom.repo.tld/custom_fork.git Options: -b BRANCH, --branch=BRANCH Select which branch of kohya to checkout on new installs. -d DIR, --dir=DIR The full path you want kohya_ss installed to. - -h, --help Show this screen. + -g, --git_repo You can optionally provide a git repo to checkout for runpod installation. Useful for custom forks. + -r, --runpod Forces a runpod installation. Useful if detection fails for any reason. + -h, --help Show this screen. EOF } # Variables defined before the getopts loop, so we have sane default values. DIR="/workspace/kohya_ss" BRANCH="dev" -REPO="https://github.com/bmaltais/kohya_ss.git" +GIT_REPO="https://github.com/bmaltais/kohya_ss.git" +RUNPOD=false -while getopts "b:d:-:" opt; do +while getopts "b:d:g:r-:" opt; do # support long options: https://stackoverflow.com/a/28466267/519360 if [ "$opt" = "-" ]; then # long option: reformulate OPT and OPTARG opt="${OPTARG%%=*}" # extract long option name @@ -36,7 +39,8 @@ while getopts "b:d:-:" opt; do # note the leading colon b | branch) BRANCH="$OPTARG" ;; d | dir) DIR="$OPTARG" ;; - r | repo) REPO="$OPTARG" ;; + g | git-repo) GIT_REPO="$OPTARG" ;; + r | runpod) RUNPOD=true ;; h) display_help && exit 0 ;; *) display_help && exit 0 ;; esac @@ -54,6 +58,10 @@ if [[ "$OSTYPE" == "linux-gnu"* ]]; then root=false fi + if env_var_exists RUNPOD_POD_ID || env_var_exists RUNPOD_API_KEY; then + RUNPOD=true + fi + env_var_exists() { local env_var= env_var=$(declare -p "$1") @@ -110,17 +118,17 @@ if [[ "$OSTYPE" == "linux-gnu"* ]]; then } # This is the pre-install work for a kohya installation on a runpod - if env_var_exists RUNPOD_POD_ID || env_var_exists RUNPOD_API_KEY; then + if [ "$RUNPOD" = true ]; then if [ -d "$VENV_DIR" ]; then echo "Pre-existing installation on a runpod detected." export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:"$VENV_DIR"/lib/python3.10/site-packages/tensorrt/ export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:"$VENV_DIR"/lib/python3.10/site-packages/nvidia/cuda_runtime/lib/ cd "$DIR" || exit 1 - sed -i "s/interface.launch(\*\*launch_kwargs)/interface.launch(\*\*launch_kwargs,share=True)/g" kohya_gui.py + sed -i "s/interface.launch(\*\*launch_kwargs)/interface.launch(\*\*launch_kwargs,share=True)/g" ./kohya_gui.py else echo "Clean installation on a runpod detected." cd "$BASE_DIR" || exit 1 - git clone "$REPO" + git clone "$GIT_REPO" cd "$DIR" || exit 1 git checkout "$BRANCH" fi @@ -185,7 +193,7 @@ if [[ "$OSTYPE" == "linux-gnu"* ]]; then pip install -U -I --no-deps https://github.com/C43H66N12O12S2/stable-diffusion-webui/releases/download/linux/xformers-0.0.14.dev0-cp310-cp310-linux_x86_64.whl # We need this extra package and setup if we are running in a runpod - if env_var_exists RUNPOD_POD_ID || env_var_exists RUNPOD_API_KEY; then + if [ "$RUNPOD" = true ]; then pip install tensorrt ln -s "$VENV_DIR/lib/python3.10/site-packages/tensorrt/libnvinfer_plugin.so.8" \ "$VENV_DIR/lib/python3.10/site-packages/tensorrt/libnvinfer_plugin.so.7" From 981f634c31ae3ff8fc1bd23a1e87084d583bf8f9 Mon Sep 17 00:00:00 2001 From: jstayco <127801635+jstayco@users.noreply.github.com> Date: Wed, 29 Mar 2023 09:55:04 -0700 Subject: [PATCH 08/35] Removed superfluous comment --- setup.sh | 1 - 1 file changed, 1 deletion(-) diff --git a/setup.sh b/setup.sh index 9af9320..afaed56 100755 --- a/setup.sh +++ b/setup.sh @@ -36,7 +36,6 @@ while getopts "b:d:g:r-:" opt; do OPTARG="${OPTARG#=}" # if long option argument, remove assigning `=` fi case $opt in - # note the leading colon b | branch) BRANCH="$OPTARG" ;; d | dir) DIR="$OPTARG" ;; g | git-repo) GIT_REPO="$OPTARG" ;; From 20c525bd59c282855310b9f34b506e4ee0fc6cd8 Mon Sep 17 00:00:00 2001 From: JSTayco Date: Wed, 29 Mar 2023 10:53:15 -0700 Subject: [PATCH 09/35] Enable more arbitrary install locations Default to master branch (user can override with an argument), update the repo if we find a git folder but no venv folder (indicating blank env), rename BASE_DIR to PARENT_DIR to be more obvious, enable PARENT_DIR to account for an arbitrary amount of folders. --- setup.sh | 20 ++++++++++++++------ 1 file changed, 14 insertions(+), 6 deletions(-) diff --git a/setup.sh b/setup.sh index 9af9320..7965fbd 100755 --- a/setup.sh +++ b/setup.sh @@ -24,7 +24,7 @@ EOF # Variables defined before the getopts loop, so we have sane default values. DIR="/workspace/kohya_ss" -BRANCH="dev" +BRANCH="master" GIT_REPO="https://github.com/bmaltais/kohya_ss.git" RUNPOD=false @@ -48,7 +48,7 @@ done shift $((OPTIND - 1)) # This must be set after the getopts loop to account for $DIR changes. -BASE_DIR="$(echo "$DIR" | cut -d "/" -f2)" +PARENT_DIR="$(dirname "${DIR}")" VENV_DIR="$DIR/venv" if [[ "$OSTYPE" == "linux-gnu"* ]]; then @@ -127,10 +127,18 @@ if [[ "$OSTYPE" == "linux-gnu"* ]]; then sed -i "s/interface.launch(\*\*launch_kwargs)/interface.launch(\*\*launch_kwargs,share=True)/g" ./kohya_gui.py else echo "Clean installation on a runpod detected." - cd "$BASE_DIR" || exit 1 - git clone "$GIT_REPO" - cd "$DIR" || exit 1 - git checkout "$BRANCH" + cd "$PARENT_DIR" || exit 1 + if [ ! -d "$DIR/.git" ]; then + echo "Cloning $GIT_REPO." + git clone "$GIT_REPO" + cd "$DIR" || exit 1 + git checkout "$BRANCH" + else + cd "$DIR" || exit 1 + echo "git repo detected. Attempting tp update repo instead." + echo "Updating: $GIT_REPO" + git pull "$GIT_REPO" + fi fi fi From 9a9976bb1c71af7c4dea2f5c1fc5653d1771b8ad Mon Sep 17 00:00:00 2001 From: JSTayco Date: Wed, 29 Mar 2023 11:11:10 -0700 Subject: [PATCH 10/35] Move env_var_exist first usage after the function is defined --- setup.sh | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/setup.sh b/setup.sh index 50ab23a..be8cb56 100755 --- a/setup.sh +++ b/setup.sh @@ -57,10 +57,6 @@ if [[ "$OSTYPE" == "linux-gnu"* ]]; then root=false fi - if env_var_exists RUNPOD_POD_ID || env_var_exists RUNPOD_API_KEY; then - RUNPOD=true - fi - env_var_exists() { local env_var= env_var=$(declare -p "$1") @@ -116,6 +112,10 @@ if [[ "$OSTYPE" == "linux-gnu"* ]]; then fi } + if env_var_exists RUNPOD_POD_ID || env_var_exists RUNPOD_API_KEY; then + RUNPOD=true + fi + # This is the pre-install work for a kohya installation on a runpod if [ "$RUNPOD" = true ]; then if [ -d "$VENV_DIR" ]; then From 403c2b051ce14a38b7996b5b229283c2b546817e Mon Sep 17 00:00:00 2001 From: JSTayco Date: Wed, 29 Mar 2023 11:42:36 -0700 Subject: [PATCH 11/35] More output and fixed package detection on Ubuntu Ubuntu dpkg detection fixed and more output for detected distros. --- setup.sh | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/setup.sh b/setup.sh index be8cb56..4bf0ca4 100755 --- a/setup.sh +++ b/setup.sh @@ -143,10 +143,12 @@ if [[ "$OSTYPE" == "linux-gnu"* ]]; then distro=get_distro_name family=get_distro_family + + echo "Installing Python TK if not found on the system." + if "$distro" | grep -qi "Ubuntu" || "$family" | grep -qi "Ubuntu"; then echo "Ubuntu detected." - echo "Installing Python TK if not found on the system." - if [ ! $(dpkg-query -W -f='${Status}' python3-tk 2>/dev/null | grep -c "ok installed") -eq 0 ]; then + if [ $(dpkg-query -W -f='${Status}' python3-tk 2>/dev/null | grep -c "ok installed") = 0 ]; then if [ root = true ]; then apt update -y && apt install -y python3-tk else @@ -157,6 +159,7 @@ if [[ "$OSTYPE" == "linux-gnu"* ]]; then echo "Python TK found! Skipping install!" fi elif "$distro" | grep -Eqi "Fedora|CentOS|Redhat"; then + echo "Redhat or Redhat base detected." if ! rpm -qa | grep -qi python3-tkinter; then if [ root = true ]; then dnf install python3-tkinter -y @@ -166,6 +169,7 @@ if [[ "$OSTYPE" == "linux-gnu"* ]]; then fi fi elif "$distro" | grep -Eqi "arch" || "$family" | grep -qi "arch"; then + echo "Arch Linux or Arch base detected." if ! pacman -Qi tk >/dev/null; then if [ root = true ]; then pacman --noconfirm -S tk @@ -175,6 +179,7 @@ if [[ "$OSTYPE" == "linux-gnu"* ]]; then fi fi elif "$distro" | grep -Eqi "opensuse" || "$family" | grep -qi "opensuse"; then + echo "OpenSUSE detected." if ! rpm -qa | grep -qi python-tk; then if [ root = true ]; then zypper install -y python-tk From a99ae6cff769ba5c500206210c02b64d863cf8ba Mon Sep 17 00:00:00 2001 From: JSTayco Date: Wed, 29 Mar 2023 12:07:00 -0700 Subject: [PATCH 12/35] Fixed root detection and made more robust --- setup.sh | 18 +++++++++++------- 1 file changed, 11 insertions(+), 7 deletions(-) diff --git a/setup.sh b/setup.sh index 4bf0ca4..6bdb21b 100755 --- a/setup.sh +++ b/setup.sh @@ -52,9 +52,13 @@ VENV_DIR="$DIR/venv" if [[ "$OSTYPE" == "linux-gnu"* ]]; then # Check if root or sudo - root=true - if [ "$EUID" -ne 0 ]; then - root=false + root=false + if [ "$EUID" = 0 ]; then + root=true + elif command -v id >/dev/null && [ "$(id -u)" = 0 ]; then + root=true + elif [ "$UID" = 0 ]; then + root=true fi env_var_exists() { @@ -149,7 +153,7 @@ if [[ "$OSTYPE" == "linux-gnu"* ]]; then if "$distro" | grep -qi "Ubuntu" || "$family" | grep -qi "Ubuntu"; then echo "Ubuntu detected." if [ $(dpkg-query -W -f='${Status}' python3-tk 2>/dev/null | grep -c "ok installed") = 0 ]; then - if [ root = true ]; then + if [ "$root" = true ]; then apt update -y && apt install -y python3-tk else echo "This script needs to be run as root or via sudo to install packages." @@ -161,7 +165,7 @@ if [[ "$OSTYPE" == "linux-gnu"* ]]; then elif "$distro" | grep -Eqi "Fedora|CentOS|Redhat"; then echo "Redhat or Redhat base detected." if ! rpm -qa | grep -qi python3-tkinter; then - if [ root = true ]; then + if [ "$root" = true ]; then dnf install python3-tkinter -y else echo "This script needs to be run as root or via sudo to install packages." @@ -171,7 +175,7 @@ if [[ "$OSTYPE" == "linux-gnu"* ]]; then elif "$distro" | grep -Eqi "arch" || "$family" | grep -qi "arch"; then echo "Arch Linux or Arch base detected." if ! pacman -Qi tk >/dev/null; then - if [ root = true ]; then + if [ "$root" = true ]; then pacman --noconfirm -S tk else echo "This script needs to be run as root or via sudo to install packages." @@ -181,7 +185,7 @@ if [[ "$OSTYPE" == "linux-gnu"* ]]; then elif "$distro" | grep -Eqi "opensuse" || "$family" | grep -qi "opensuse"; then echo "OpenSUSE detected." if ! rpm -qa | grep -qi python-tk; then - if [ root = true ]; then + if [ "$root" = true ]; then zypper install -y python-tk else echo "This script needs to be run as root or via sudo to install packages." From ee1ac64034322f0732087c4e28f521a563cd7ba4 Mon Sep 17 00:00:00 2001 From: JSTayco Date: Wed, 29 Mar 2023 12:23:43 -0700 Subject: [PATCH 13/35] Added setup.sh -h to the README Some environments can't run setup.sh interactively. This will be convenient for those users. --- README.md | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/README.md b/README.md index 0a866b2..1ffc7e2 100644 --- a/README.md +++ b/README.md @@ -68,6 +68,26 @@ cd kohya_ss ./setup.sh ``` +Setup.sh help included here: + +```bash +Kohya_SS Installation Script for POSIX operating systems. + +The following options are useful in a runpod environment, +but will not affect a local machine install. + +Usage: + setup.sh -b dev -d /workspace/kohya_ss -g https://mycustom.repo.tld/custom_fork.git + setup.sh --branch=dev --dir=/workspace/kohya_ss --git-repo=https://mycustom.repo.tld/custom_fork.git + +Options: + -b BRANCH, --branch=BRANCH Select which branch of kohya to checkout on new installs. + -d DIR, --dir=DIR The full path you want kohya_ss installed to. + -g, --git_repo You can optionally provide a git repo to checkout for runpod installation. Useful for custom forks. + -r, --runpod Forces a runpod installation. Useful if detection fails for any reason. + -h, --help Show this screen. +``` + During the accelerate config screen after running the script answer "This machine", "None", "No" for the remaining questions. These are the same answers as the Windows install. From 71c9459db23a02e210607d0070f50082dfdee909 Mon Sep 17 00:00:00 2001 From: JSTayco Date: Wed, 29 Mar 2023 13:26:16 -0700 Subject: [PATCH 14/35] Non-interactive mode, new warning, new default config for accel We now warn the user with a nicely formatted message if they have less than 10Gb space free and offer a 10s window to cancel operations. We now try to configure accelerate with no user input by default, but allow an override. --- README.md | 3 +- config_files/accelerate/default_config.yaml | 22 +++++++++ setup.sh | 53 +++++++++++++++++++-- 3 files changed, 74 insertions(+), 4 deletions(-) create mode 100644 config_files/accelerate/default_config.yaml diff --git a/README.md b/README.md index 1ffc7e2..9a04735 100644 --- a/README.md +++ b/README.md @@ -85,10 +85,11 @@ Options: -d DIR, --dir=DIR The full path you want kohya_ss installed to. -g, --git_repo You can optionally provide a git repo to checkout for runpod installation. Useful for custom forks. -r, --runpod Forces a runpod installation. Useful if detection fails for any reason. + -i, --interactive Interactively configure accelerate instead of using default config file. -h, --help Show this screen. ``` -During the accelerate config screen after running the script answer "This machine", "None", "No" for the remaining questions. +If you are using the interactive mode, our default values for the accelerate config screen after running the script answer "This machine", "None", "No" for the remaining questions. These are the same answers as the Windows install. ### Windows diff --git a/config_files/accelerate/default_config.yaml b/config_files/accelerate/default_config.yaml new file mode 100644 index 0000000..a31ddd0 --- /dev/null +++ b/config_files/accelerate/default_config.yaml @@ -0,0 +1,22 @@ +command_file: null +commands: null +compute_environment: LOCAL_MACHINE +deepspeed_config: {} +distributed_type: 'NO' +downcast_bf16: 'no' +dynamo_backend: 'NO' +fsdp_config: {} +gpu_ids: all +machine_rank: 0 +main_process_ip: null +main_process_port: null +main_training_function: main +megatron_lm_config: {} +mixed_precision: 'no' +num_machines: 1 +num_processes: 1 +rdzv_backend: static +same_network: true +tpu_name: null +tpu_zone: null +use_cpu: false diff --git a/setup.sh b/setup.sh index 6bdb21b..83a92d8 100755 --- a/setup.sh +++ b/setup.sh @@ -18,6 +18,7 @@ Options: -d DIR, --dir=DIR The full path you want kohya_ss installed to. -g, --git_repo You can optionally provide a git repo to checkout for runpod installation. Useful for custom forks. -r, --runpod Forces a runpod installation. Useful if detection fails for any reason. + -i, --interactive Interactively configure accelerate instead of using default config file. -h, --help Show this screen. EOF } @@ -27,8 +28,9 @@ DIR="/workspace/kohya_ss" BRANCH="master" GIT_REPO="https://github.com/bmaltais/kohya_ss.git" RUNPOD=false +INTERACTIVE=false -while getopts "b:d:g:r-:" opt; do +while getopts "b:d:g:ir-:" opt; do # support long options: https://stackoverflow.com/a/28466267/519360 if [ "$opt" = "-" ]; then # long option: reformulate OPT and OPTARG opt="${OPTARG%%=*}" # extract long option name @@ -39,6 +41,7 @@ while getopts "b:d:g:r-:" opt; do b | branch) BRANCH="$OPTARG" ;; d | dir) DIR="$OPTARG" ;; g | git-repo) GIT_REPO="$OPTARG" ;; + i | interactive) INTERACTIVE=true ;; r | runpod) RUNPOD=true ;; h) display_help && exit 0 ;; *) display_help && exit 0 ;; @@ -116,10 +119,29 @@ if [[ "$OSTYPE" == "linux-gnu"* ]]; then fi } + # This checks for free space on the installation drive and returns that in Gb. + size_available() { + local FREESPACEINKB="$(df -Pk "$DIR" | sed 1d | grep -v used | awk '{ print $4 "\t" }')" + local FREESPACEINGB=$((FREESPACEINKB / 1024 / 1024)) + echo "$FREESPACEINGB" + } + if env_var_exists RUNPOD_POD_ID || env_var_exists RUNPOD_API_KEY; then RUNPOD=true fi + # Offer a warning and opportunity to cancel the installation if < 10Gb of Free Space detected + if [ "$(size_available)" -lt 10 ]; then + echo "You have less than 10Gb of free space. This installation may fail." + MSGTIMEOUT=10 # In seconds + MESSAGE="Continuing in..." + echo "Press control-c to cancel the installation." + for ((i = $MSGTIMEOUT; i >= 0; i--)); do + printf "\r${MESSAGE} %ss. " "${i}" + sleep 1 + done + fi + # This is the pre-install work for a kohya installation on a runpod if [ "$RUNPOD" = true ]; then if [ -d "$VENV_DIR" ]; then @@ -221,6 +243,33 @@ if [[ "$OSTYPE" == "linux-gnu"* ]]; then export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$VENV_DIR/lib/python3.10/site-packages/tensorrt/" export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$VENV_DIR/lib/python3.10/site-packages/nvidia/cuda_runtime/lib/" + # Attempt to non-interactively install a default accelerate config file unless specified otherwise. + # Documentation for order of precedence locations for configuration file for automated installation: + # https://huggingface.co/docs/accelerate/basic_tutorials/launch#custom-configurations + if [ "$INTERACTIVE" = true ]; then + accelerate config + else + if env_var_exists HF_HOME; then + if [ ! -f "$HF_HOME/accelerate/default_config.yaml" ]; then + mkdir -p "$HF_HOME/accelerate/" && + cp ./config_files/accelerate/default_config.yaml "$HF_HOME/accelerate/default_config.yaml" + fi + elif env_var_exists XDG_CACHE_HOME; then + if [ ! -f "$XDG_CACHE_HOME/huggingface/accelerate" ]; then + mkdir -p "$XDG_CACHE_HOME/huggingface/accelerate" && + cp ./config_files/accelerate/default_config.yaml "$XDG_CACHE_HOME/huggingface/accelerate/default_config.yaml" + fi + elif env_var_exists HOME; then + if [ ! -f "$HOME/.cache/huggingface/accelerate" ]; then + mkdir -p "$HOME/.cache/huggingface/accelerate" && + cp ./config_files/accelerate/default_config.yaml "$HOME/.cache/huggingface/accelerate/default_config.yaml" + fi + else + echo "Could not place the accelerate configuration file. Please configure manually." + accelerate config + fi + fi + # This is a non-interactive environment, so just directly call gui.sh after all setup steps are complete. if command -v bash >/dev/null; then bash "$DIR"/gui.sh @@ -230,8 +279,6 @@ if [[ "$OSTYPE" == "linux-gnu"* ]]; then fi fi - accelerate config - echo -e "Setup finished! Run \e[0;92m./gui.sh\e[0m to start." elif [[ "$OSTYPE" == "darwin"* ]]; then # The initial setup script to prep the environment on macOS From 094528c7cd449ef450053ccdce885a5770462bfe Mon Sep 17 00:00:00 2001 From: JSTayco Date: Wed, 29 Mar 2023 13:35:10 -0700 Subject: [PATCH 15/35] Inform user of config copy Better inform the user what is happening. Upon successful configuration file copy operation, we notify the user where that config is located. Also added small sleep step before calling accelerate config to give the user a chance to read the message. --- setup.sh | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/setup.sh b/setup.sh index 83a92d8..564000b 100755 --- a/setup.sh +++ b/setup.sh @@ -252,20 +252,24 @@ if [[ "$OSTYPE" == "linux-gnu"* ]]; then if env_var_exists HF_HOME; then if [ ! -f "$HF_HOME/accelerate/default_config.yaml" ]; then mkdir -p "$HF_HOME/accelerate/" && - cp ./config_files/accelerate/default_config.yaml "$HF_HOME/accelerate/default_config.yaml" + cp ./config_files/accelerate/default_config.yaml "$HF_HOME/accelerate/default_config.yaml" && + echo "Copied accelerate config file to: $HF_HOME/accelerate/default_config.yaml" fi elif env_var_exists XDG_CACHE_HOME; then if [ ! -f "$XDG_CACHE_HOME/huggingface/accelerate" ]; then mkdir -p "$XDG_CACHE_HOME/huggingface/accelerate" && - cp ./config_files/accelerate/default_config.yaml "$XDG_CACHE_HOME/huggingface/accelerate/default_config.yaml" + cp ./config_files/accelerate/default_config.yaml "$XDG_CACHE_HOME/huggingface/accelerate/default_config.yaml" && + echo "Copied accelerate config file to: $XDG_CACHE_HOME/huggingface/accelerate/default_config.yaml" fi elif env_var_exists HOME; then if [ ! -f "$HOME/.cache/huggingface/accelerate" ]; then mkdir -p "$HOME/.cache/huggingface/accelerate" && - cp ./config_files/accelerate/default_config.yaml "$HOME/.cache/huggingface/accelerate/default_config.yaml" + cp ./config_files/accelerate/default_config.yaml "$HOME/.cache/huggingface/accelerate/default_config.yaml" && + echo "Copying accelerate config file to: $HOME/.cache/huggingface/accelerate/default_config.yaml" fi else echo "Could not place the accelerate configuration file. Please configure manually." + sleep 2 accelerate config fi fi From a58b3b616aa382f8c0963d36321b86c418d36502 Mon Sep 17 00:00:00 2001 From: JSTayco Date: Wed, 29 Mar 2023 14:41:17 -0700 Subject: [PATCH 16/35] Add -p switch to expose public URL This adds the ability for an extra switch to expose a public URL. We default to private. --- README.md | 1 + setup.sh | 16 +++++++++++++--- 2 files changed, 14 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 9a04735..a81de9b 100644 --- a/README.md +++ b/README.md @@ -84,6 +84,7 @@ Options: -b BRANCH, --branch=BRANCH Select which branch of kohya to checkout on new installs. -d DIR, --dir=DIR The full path you want kohya_ss installed to. -g, --git_repo You can optionally provide a git repo to checkout for runpod installation. Useful for custom forks. + -p, --public Expose public URL in runpod mode. Won't have an effect in other modes. -r, --runpod Forces a runpod installation. Useful if detection fails for any reason. -i, --interactive Interactively configure accelerate instead of using default config file. -h, --help Show this screen. diff --git a/setup.sh b/setup.sh index 564000b..ffa9236 100755 --- a/setup.sh +++ b/setup.sh @@ -17,6 +17,7 @@ Options: -b BRANCH, --branch=BRANCH Select which branch of kohya to checkout on new installs. -d DIR, --dir=DIR The full path you want kohya_ss installed to. -g, --git_repo You can optionally provide a git repo to checkout for runpod installation. Useful for custom forks. + -p, --public Expose public URL in runpod mode. Won't have an effect in other modes. -r, --runpod Forces a runpod installation. Useful if detection fails for any reason. -i, --interactive Interactively configure accelerate instead of using default config file. -h, --help Show this screen. @@ -29,6 +30,7 @@ BRANCH="master" GIT_REPO="https://github.com/bmaltais/kohya_ss.git" RUNPOD=false INTERACTIVE=false +PUBLIC=false while getopts "b:d:g:ir-:" opt; do # support long options: https://stackoverflow.com/a/28466267/519360 @@ -42,6 +44,7 @@ while getopts "b:d:g:ir-:" opt; do d | dir) DIR="$OPTARG" ;; g | git-repo) GIT_REPO="$OPTARG" ;; i | interactive) INTERACTIVE=true ;; + p | public) PUBLIC=true ;; r | runpod) RUNPOD=true ;; h) display_help && exit 0 ;; *) display_help && exit 0 ;; @@ -149,7 +152,6 @@ if [[ "$OSTYPE" == "linux-gnu"* ]]; then export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:"$VENV_DIR"/lib/python3.10/site-packages/tensorrt/ export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:"$VENV_DIR"/lib/python3.10/site-packages/nvidia/cuda_runtime/lib/ cd "$DIR" || exit 1 - sed -i "s/interface.launch(\*\*launch_kwargs)/interface.launch(\*\*launch_kwargs,share=True)/g" ./kohya_gui.py else echo "Clean installation on a runpod detected." cd "$PARENT_DIR" || exit 1 @@ -276,10 +278,18 @@ if [[ "$OSTYPE" == "linux-gnu"* ]]; then # This is a non-interactive environment, so just directly call gui.sh after all setup steps are complete. if command -v bash >/dev/null; then - bash "$DIR"/gui.sh + if [ "$PUBLIC" = false ]; then + bash "$DIR"/gui.sh + else + bash "$DIR"/gui.sh --share + fi else # This shouldn't happen, but we're going to try to help. - sh "$DIR"/gui.sh + if [ "$PUBLIC" = false ]; then + sh "$DIR"/gui.sh + else + sh "$DIR"/gui.sh --share + fi fi fi From 1f67aaa43b2864d9352a6d1283765e8ee46d2101 Mon Sep 17 00:00:00 2001 From: JSTayco Date: Wed, 29 Mar 2023 14:47:10 -0700 Subject: [PATCH 17/35] More robust df check --- setup.sh | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/setup.sh b/setup.sh index ffa9236..8f5344d 100755 --- a/setup.sh +++ b/setup.sh @@ -124,7 +124,19 @@ if [[ "$OSTYPE" == "linux-gnu"* ]]; then # This checks for free space on the installation drive and returns that in Gb. size_available() { - local FREESPACEINKB="$(df -Pk "$DIR" | sed 1d | grep -v used | awk '{ print $4 "\t" }')" + local folder + if [ -d "$DIR" ]; then + folder="$DIR" + elif [ -d "$PARENT_DIR" ]; then + folder="$PARENT_DIR" + elif [ -d "$(echo "$DIR" | cut -d "/" -f2)" ]; then + folder="$(echo "$DIR" | cut -d "/" -f2)" + else + echo "We are assuming a root drive install for space-checking purposes." + folder='/' + fi + + local FREESPACEINKB="$(df -Pk "$folder" | sed 1d | grep -v used | awk '{ print $4 "\t" }')" local FREESPACEINGB=$((FREESPACEINKB / 1024 / 1024)) echo "$FREESPACEINGB" } From c490764c7ee56be2c5e502f21db5cb1f4acfcc11 Mon Sep 17 00:00:00 2001 From: JSTayco Date: Wed, 29 Mar 2023 15:07:04 -0700 Subject: [PATCH 18/35] Simplify environment variable check --- setup.sh | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/setup.sh b/setup.sh index 8f5344d..83bb992 100755 --- a/setup.sh +++ b/setup.sh @@ -67,11 +67,12 @@ if [[ "$OSTYPE" == "linux-gnu"* ]]; then root=true fi + # Checks to see if variable is set and non-empty. env_var_exists() { - local env_var= - env_var=$(declare -p "$1") - if ! [[ -v $1 && $env_var =~ ^declare\ -x ]]; then + if [[ ! -v "$1" ]] || [[ -z "$1" ]]; then return 1 + else + return 0 fi } From dfe96a581d78196e06fbbbcad3850a083ad4b4f1 Mon Sep 17 00:00:00 2001 From: JSTayco Date: Wed, 29 Mar 2023 15:09:42 -0700 Subject: [PATCH 19/35] Remove hard-coded config file path --- setup.sh | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/setup.sh b/setup.sh index 83bb992..152eb67 100755 --- a/setup.sh +++ b/setup.sh @@ -267,19 +267,19 @@ if [[ "$OSTYPE" == "linux-gnu"* ]]; then if env_var_exists HF_HOME; then if [ ! -f "$HF_HOME/accelerate/default_config.yaml" ]; then mkdir -p "$HF_HOME/accelerate/" && - cp ./config_files/accelerate/default_config.yaml "$HF_HOME/accelerate/default_config.yaml" && + cp "$DIR/config_files/accelerate/default_config.yaml" "$HF_HOME/accelerate/default_config.yaml" && echo "Copied accelerate config file to: $HF_HOME/accelerate/default_config.yaml" fi elif env_var_exists XDG_CACHE_HOME; then if [ ! -f "$XDG_CACHE_HOME/huggingface/accelerate" ]; then mkdir -p "$XDG_CACHE_HOME/huggingface/accelerate" && - cp ./config_files/accelerate/default_config.yaml "$XDG_CACHE_HOME/huggingface/accelerate/default_config.yaml" && + cp "$DIR/config_files/accelerate/default_config.yaml" "$XDG_CACHE_HOME/huggingface/accelerate/default_config.yaml" && echo "Copied accelerate config file to: $XDG_CACHE_HOME/huggingface/accelerate/default_config.yaml" fi elif env_var_exists HOME; then if [ ! -f "$HOME/.cache/huggingface/accelerate" ]; then mkdir -p "$HOME/.cache/huggingface/accelerate" && - cp ./config_files/accelerate/default_config.yaml "$HOME/.cache/huggingface/accelerate/default_config.yaml" && + cp "$DIR/config_files/accelerate/default_config.yaml" "$HOME/.cache/huggingface/accelerate/default_config.yaml" && echo "Copying accelerate config file to: $HOME/.cache/huggingface/accelerate/default_config.yaml" fi else From 38aab1c30a7038cb38d16f1e1643777fe733f463 Mon Sep 17 00:00:00 2001 From: jstayco <127801635+jstayco@users.noreply.github.com> Date: Wed, 29 Mar 2023 15:30:41 -0700 Subject: [PATCH 20/35] Added forgotten getopt letter --- setup.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/setup.sh b/setup.sh index 152eb67..6cf0a5b 100755 --- a/setup.sh +++ b/setup.sh @@ -32,7 +32,7 @@ RUNPOD=false INTERACTIVE=false PUBLIC=false -while getopts "b:d:g:ir-:" opt; do +while getopts "b:d:g:ipr-:" opt; do # support long options: https://stackoverflow.com/a/28466267/519360 if [ "$opt" = "-" ]; then # long option: reformulate OPT and OPTARG opt="${OPTARG%%=*}" # extract long option name From bff107878b517a220c8eb5b126fa27fdeb2d71fc Mon Sep 17 00:00:00 2001 From: JSTayco Date: Wed, 29 Mar 2023 16:05:57 -0700 Subject: [PATCH 21/35] Default install location is now environment-based Better default install locations and updated the README to reflect. --- README.md | 8 ++++++++ setup.sh | 54 ++++++++++++++++++++++++++++++++++++++---------------- 2 files changed, 46 insertions(+), 16 deletions(-) diff --git a/README.md b/README.md index a81de9b..dca0b1d 100644 --- a/README.md +++ b/README.md @@ -11,6 +11,7 @@ If you run on Linux and would like to use the GUI, there is now a port of it as - [Linux/macOS](#linux-and-macos-dependencies) - [Installation](#installation) - [Linux/macOS](#linux-and-macos) + - [Default Install Locations](#install-location) - [Windows](#windows) - [CUDNN 8.6](#optional--cudnn-86) - [Upgrading](#upgrading) @@ -90,6 +91,13 @@ Options: -h, --help Show this screen. ``` +#### Install location + +The default install location for Linux is `/opt/kohya_ss`. If /opt is not writeable, the fallback is `$HOME/kohya_ss`. Lastly, if all else fails it will simply install to the current folder you are in. + +On macOS and other non-Linux machines, it will default install to `$HOME/kohya_ss` followed by where you're currently at if there's no access to $HOME. +You can override this behavior by specifying an install directory with the -d option. + If you are using the interactive mode, our default values for the accelerate config screen after running the script answer "This machine", "None", "No" for the remaining questions. These are the same answers as the Windows install. diff --git a/setup.sh b/setup.sh index 152eb67..112d942 100755 --- a/setup.sh +++ b/setup.sh @@ -24,15 +24,50 @@ Options: EOF } +# Checks to see if variable is set and non-empty. +# This is defined first, so we can use the function for some default variable values +env_var_exists() { + if [[ ! -v "$1" ]] || [[ -z "$1" ]]; then + return 1 + else + return 0 + fi +} + +# Need RUNPOD to have a default value before first access +RUNPOD=false +if env_var_exists RUNPOD_POD_ID || env_var_exists RUNPOD_API_KEY; then + RUNPOD=true +fi + # Variables defined before the getopts loop, so we have sane default values. -DIR="/workspace/kohya_ss" +# Default installation locations based on OS and environment +if [[ "$OSTYPE" == "linux-gnu"* ]]; then + if [ "$RUNPOD" = true ]; then + DIR="/workspace/kohya_ss" + elif [ -w "/opt" ]; then + DIR="/opt/kohya_ss" + elif env_var_exists HOME; then + DIR="$HOME/kohya_ss" + else + # The last fallback is simply PWD + DIR="$(PWD)" + fi +else + if env_var_exists HOME; then + DIR="$HOME/kohya_ss" + else + # The last fallback is simply PWD + DIR="$(PWD)" + fi +fi + BRANCH="master" GIT_REPO="https://github.com/bmaltais/kohya_ss.git" -RUNPOD=false INTERACTIVE=false PUBLIC=false -while getopts "b:d:g:ir-:" opt; do +while getopts "b:d:g:ipr-:" opt; do # support long options: https://stackoverflow.com/a/28466267/519360 if [ "$opt" = "-" ]; then # long option: reformulate OPT and OPTARG opt="${OPTARG%%=*}" # extract long option name @@ -67,15 +102,6 @@ if [[ "$OSTYPE" == "linux-gnu"* ]]; then root=true fi - # Checks to see if variable is set and non-empty. - env_var_exists() { - if [[ ! -v "$1" ]] || [[ -z "$1" ]]; then - return 1 - else - return 0 - fi - } - get_distro_name() { local line if [ -f /etc/os-release ]; then @@ -142,10 +168,6 @@ if [[ "$OSTYPE" == "linux-gnu"* ]]; then echo "$FREESPACEINGB" } - if env_var_exists RUNPOD_POD_ID || env_var_exists RUNPOD_API_KEY; then - RUNPOD=true - fi - # Offer a warning and opportunity to cancel the installation if < 10Gb of Free Space detected if [ "$(size_available)" -lt 10 ]; then echo "You have less than 10Gb of free space. This installation may fail." From 90e34cc6f72ffc39ebea23b5624dea4ccecffe77 Mon Sep 17 00:00:00 2001 From: jstayco <127801635+jstayco@users.noreply.github.com> Date: Thu, 30 Mar 2023 16:20:13 -0700 Subject: [PATCH 22/35] Small improvement to an stdout line. Just a typo fix and added a word in an echo statement. --- setup.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/setup.sh b/setup.sh index 112d942..c24d48b 100755 --- a/setup.sh +++ b/setup.sh @@ -197,7 +197,7 @@ if [[ "$OSTYPE" == "linux-gnu"* ]]; then git checkout "$BRANCH" else cd "$DIR" || exit 1 - echo "git repo detected. Attempting tp update repo instead." + echo "git repo detected. Attempting to update repository instead." echo "Updating: $GIT_REPO" git pull "$GIT_REPO" fi From 6e7e25cba927d97ef599a48a7de567f3d5687e6c Mon Sep 17 00:00:00 2001 From: JSTayco Date: Thu, 30 Mar 2023 18:20:42 -0700 Subject: [PATCH 23/35] Big update: Verbosity, space check skip, Huge reorg to account for macOS and other non-Linux OSs. Verbosity levels 1-3 added (used by FDs 3-5). Simplified code by creating more shared functions. --- setup.sh | 291 ++++++++++++++++++++++++++++++++++++++----------------- 1 file changed, 201 insertions(+), 90 deletions(-) diff --git a/setup.sh b/setup.sh index 112d942..6a5c392 100755 --- a/setup.sh +++ b/setup.sh @@ -6,21 +6,26 @@ display_help() { cat <&2" #Don't change anything higher than the maximum verbosity allowed. +done + +for v in $( #From the verbosity level one higher than requested, through the maximum; + seq $((VERBOSITY + 1)) $MAXVERBOSITY +); do + (("$v" > "2")) && eval exec "$v>/dev/null" #Redirect these to bitbucket, provided that they don't match stdout and stderr. +done + +# Example of how to use the verbosity levels. +# printf "%s\n" "This message is seen at verbosity level 1 and above." >&3 +# printf "%s\n" "This message is seen at verbosity level 2 and above." >&4 +# printf "%s\n" "This message is seen at verbosity level 3 and above." >&5 + +# Debug variable dump at max verbosity +echo "BRANCH: $BRANCH +DIR: $DIR +GIT_REPO: $GIT_REPO +INTERACTIVE: $INTERACTIVE +PUBLIC: $PUBLIC +RUNPOD: $RUNPOD +SKIP_SPACE_CHECK: $SKIP_SPACE_CHECK +VERBOSITY: $VERBOSITY" >&5 + # This must be set after the getopts loop to account for $DIR changes. PARENT_DIR="$(dirname "${DIR}")" VENV_DIR="$DIR/venv" +# Shared functions +# This checks for free space on the installation drive and returns that in Gb. +size_available() { + local folder + if [ -d "$DIR" ]; then + folder="$DIR" + elif [ -d "$PARENT_DIR" ]; then + folder="$PARENT_DIR" + elif [ -d "$(echo "$DIR" | cut -d "/" -f2)" ]; then + folder="$(echo "$DIR" | cut -d "/" -f2)" + else + echo "We are assuming a root drive install for space-checking purposes." + folder='/' + fi + + local FREESPACEINKB="$(df -Pk "$folder" | sed 1d | grep -v used | awk '{ print $4 "\t" }')" + echo "Detected available space in Kb: $FREESPACEINKB" >&5 + local FREESPACEINGB=$((FREESPACEINKB / 1024 / 1024)) + echo "$FREESPACEINGB" +} + +# The expected usage is create_symlinks $symlink $target_file +create_symlinks() { + echo "Checking symlinks now." + # Next line checks for valid symlink + if [ -L "$1" ]; then + # Check if the linked file exists and points to the expected file + if [ -e "$1" ] && [ "$(readlink "$1")" == "$2" ]; then + echo "$(basename "$1") symlink looks fine. Skipping." + else + if [ -f "$2" ]; then + echo "Broken symlink detected. Recreating $(basename "$1")." + rm "$1" && + ln -s "$2" "$1" + else + echo "$2 does not exist. Nothing to link." + fi + fi + else + echo "Linking $(basename "$1")." + ln -s "$2" "$1" + fi +} + +# Attempt to non-interactively install a default accelerate config file unless specified otherwise. +# Documentation for order of precedence locations for configuration file for automated installation: +# https://huggingface.co/docs/accelerate/basic_tutorials/launch#custom-configurations +configure_accelerate() { + echo "Source accelerate config location: $DIR/config_files/accelerate/default_config.yaml" >&3 + if [ "$INTERACTIVE" = true ]; then + accelerate config + else + if env_var_exists HF_HOME; then + if [ ! -f "$HF_HOME/accelerate/default_config.yaml" ]; then + mkdir -p "$HF_HOME/accelerate/" && + echo "Target accelerate config location: $HF_HOME/accelerate/default_config.yaml" >&3 + cp "$DIR/config_files/accelerate/default_config.yaml" "$HF_HOME/accelerate/default_config.yaml" && + echo "Copied accelerate config file to: $HF_HOME/accelerate/default_config.yaml" + fi + elif env_var_exists XDG_CACHE_HOME; then + if [ ! -f "$XDG_CACHE_HOME/huggingface/accelerate" ]; then + mkdir -p "$XDG_CACHE_HOME/huggingface/accelerate" && + echo "Target accelerate config location: $XDG_CACHE_HOME/accelerate/default_config.yaml" >&3 + cp "$DIR/config_files/accelerate/default_config.yaml" "$XDG_CACHE_HOME/huggingface/accelerate/default_config.yaml" && + echo "Copied accelerate config file to: $XDG_CACHE_HOME/huggingface/accelerate/default_config.yaml" + fi + elif env_var_exists HOME; then + if [ ! -f "$HOME/.cache/huggingface/accelerate" ]; then + mkdir -p "$HOME/.cache/huggingface/accelerate" && + echo "Target accelerate config location: $HOME/accelerate/default_config.yaml" >&3 + cp "$DIR/config_files/accelerate/default_config.yaml" "$HOME/.cache/huggingface/accelerate/default_config.yaml" && + echo "Copying accelerate config file to: $HOME/.cache/huggingface/accelerate/default_config.yaml" + fi + else + echo "Could not place the accelerate configuration file. Please configure manually." + sleep 2 + accelerate config + fi + fi +} + +# Offer a warning and opportunity to cancel the installation if < 10Gb of Free Space detected +check_storage_space() { + if [ "$SKIP_SPACE_CHECK" = false ]; then + if [ "$(size_available)" -lt 10 ]; then + echo "You have less than 10Gb of free space. This installation may fail." + MSGTIMEOUT=10 # In seconds + MESSAGE="Continuing in..." + echo "Press control-c to cancel the installation." + for ((i = $MSGTIMEOUT; i >= 0; i--)); do + printf "\r${MESSAGE} %ss. " "${i}" + sleep 1 + done + fi + fi +} + +# Start OS-specific detection and work if [[ "$OSTYPE" == "linux-gnu"* ]]; then # Check if root or sudo root=false @@ -108,6 +244,7 @@ if [[ "$OSTYPE" == "linux-gnu"* ]]; then # We search for the line starting with ID= # Then we remove the ID= prefix to get the name itself line="$(grep -Ei '^ID=' /etc/os-release)" + echo "Raw detected os-release distro line: $line" >&5 line=${line##*=} echo "$line" return 0 @@ -134,6 +271,7 @@ if [[ "$OSTYPE" == "linux-gnu"* ]]; then # This is the "type" of distro. For example, Ubuntu returns "debian". if grep -Eiq '^ID_LIKE=' /etc/os-release >/dev/null; then line="$(grep -Ei '^ID_LIKE=' /etc/os-release)" + echo "Raw detected os-release distro family line: $line" >&5 line=${line##*=} echo "$line" return 0 @@ -149,36 +287,7 @@ if [[ "$OSTYPE" == "linux-gnu"* ]]; then fi } - # This checks for free space on the installation drive and returns that in Gb. - size_available() { - local folder - if [ -d "$DIR" ]; then - folder="$DIR" - elif [ -d "$PARENT_DIR" ]; then - folder="$PARENT_DIR" - elif [ -d "$(echo "$DIR" | cut -d "/" -f2)" ]; then - folder="$(echo "$DIR" | cut -d "/" -f2)" - else - echo "We are assuming a root drive install for space-checking purposes." - folder='/' - fi - - local FREESPACEINKB="$(df -Pk "$folder" | sed 1d | grep -v used | awk '{ print $4 "\t" }')" - local FREESPACEINGB=$((FREESPACEINKB / 1024 / 1024)) - echo "$FREESPACEINGB" - } - - # Offer a warning and opportunity to cancel the installation if < 10Gb of Free Space detected - if [ "$(size_available)" -lt 10 ]; then - echo "You have less than 10Gb of free space. This installation may fail." - MSGTIMEOUT=10 # In seconds - MESSAGE="Continuing in..." - echo "Press control-c to cancel the installation." - for ((i = $MSGTIMEOUT; i >= 0; i--)); do - printf "\r${MESSAGE} %ss. " "${i}" - sleep 1 - done - fi + check_storage_space # This is the pre-install work for a kohya installation on a runpod if [ "$RUNPOD" = true ]; then @@ -192,28 +301,29 @@ if [[ "$OSTYPE" == "linux-gnu"* ]]; then cd "$PARENT_DIR" || exit 1 if [ ! -d "$DIR/.git" ]; then echo "Cloning $GIT_REPO." - git clone "$GIT_REPO" + git clone "$GIT_REPO" >&3 cd "$DIR" || exit 1 - git checkout "$BRANCH" + git checkout "$BRANCH" >&3 else cd "$DIR" || exit 1 echo "git repo detected. Attempting tp update repo instead." echo "Updating: $GIT_REPO" - git pull "$GIT_REPO" + git pull "$GIT_REPO" >&3 fi fi fi distro=get_distro_name family=get_distro_family + echo "Raw detected distro string: $distro" >&4 + echo "Raw detected distro family string: $family" >&4 echo "Installing Python TK if not found on the system." - if "$distro" | grep -qi "Ubuntu" || "$family" | grep -qi "Ubuntu"; then echo "Ubuntu detected." if [ $(dpkg-query -W -f='${Status}' python3-tk 2>/dev/null | grep -c "ok installed") = 0 ]; then if [ "$root" = true ]; then - apt update -y && apt install -y python3-tk + apt update -y >&3 && apt install -y python3-tk >&3 else echo "This script needs to be run as root or via sudo to install packages." exit 1 @@ -225,7 +335,7 @@ if [[ "$OSTYPE" == "linux-gnu"* ]]; then echo "Redhat or Redhat base detected." if ! rpm -qa | grep -qi python3-tkinter; then if [ "$root" = true ]; then - dnf install python3-tkinter -y + dnf install python3-tkinter -y >&3 else echo "This script needs to be run as root or via sudo to install packages." exit 1 @@ -235,7 +345,7 @@ if [[ "$OSTYPE" == "linux-gnu"* ]]; then echo "Arch Linux or Arch base detected." if ! pacman -Qi tk >/dev/null; then if [ "$root" = true ]; then - pacman --noconfirm -S tk + pacman --noconfirm -S tk >&3 else echo "This script needs to be run as root or via sudo to install packages." exit 1 @@ -245,7 +355,7 @@ if [[ "$OSTYPE" == "linux-gnu"* ]]; then echo "OpenSUSE detected." if ! rpm -qa | grep -qi python-tk; then if [ "$root" = true ]; then - zypper install -y python-tk + zypper install -y python-tk >&3 else echo "This script needs to be run as root or via sudo to install packages." exit 1 @@ -263,53 +373,40 @@ if [[ "$OSTYPE" == "linux-gnu"* ]]; then python3 -m venv venv source venv/bin/activate - pip install torch==1.12.1+cu116 torchvision==0.13.1+cu116 --extra-index-url https://download.pytorch.org/whl/cu116 - pip install --use-pep517 --upgrade -r requirements.txt - pip install -U -I --no-deps https://github.com/C43H66N12O12S2/stable-diffusion-webui/releases/download/linux/xformers-0.0.14.dev0-cp310-cp310-linux_x86_64.whl + + # Updating pip if there is one + echo "Checking for pip updates before Python operations." + python3 -m pip install --upgrade pip >&3 + + echo "Installing python dependencies. This could take a few minutes as it downloads files." + echo "If this operation ever runs too long, you can rerun this script in verbose mode to check." + pip install torch==1.12.1+cu116 torchvision==0.13.1+cu116 --extra-index-url https://download.pytorch.org/whl/cu116 >&3 + pip install --use-pep517 --upgrade -r requirements.txt >&3 + pip install -U -I --no-deps \ + https://github.com/C43H66N12O12S2/stable-diffusion-webui/releases/download/linux/xformers-0.0.14.dev0-cp310-cp310-linux_x86_64.whl >&3 # We need this extra package and setup if we are running in a runpod if [ "$RUNPOD" = true ]; then pip install tensorrt - ln -s "$VENV_DIR/lib/python3.10/site-packages/tensorrt/libnvinfer_plugin.so.8" \ - "$VENV_DIR/lib/python3.10/site-packages/tensorrt/libnvinfer_plugin.so.7" - ln -s "$VENV_DIR/lib/python3.10/site-packages/tensorrt/libnvinfer.so.8" \ - "$VENV_DIR/lib/python3.10/site-packages/tensorrt/libnvinfer.so.7" - ln -s "$VENV_DIR/lib/python3.10/site-packages/nvidia/cuda_runtime/lib/libcudart.so.12" \ - "$VENV_DIR/lib/python3.10/site-packages/nvidia/cuda_runtime/lib/libcudart.so.11.0" + # Symlink paths + libnvinfer_plugin_symlink="$VENV_DIR/lib/python3.10/site-packages/tensorrt/libnvinfer_plugin.so.7" + libnvinfer_symlink="$VENV_DIR/lib/python3.10/site-packages/tensorrt/libnvinfer.so.7" + libcudart_symlink="$VENV_DIR/lib/python3.10/site-packages/nvidia/cuda_runtime/lib/libcudart.so.11.0" + + #Target file paths + libnvinfer_plugin_target="$VENV_DIR/lib/python3.10/site-packages/tensorrt/libnvinfer_plugin.so.8" + libnvinfer_target="$VENV_DIR/lib/python3.10/site-packages/tensorrt/libnvinfer.so.8" + libcudart_target="$VENV_DIR/lib/python3.10/site-packages/nvidia/cuda_runtime/lib/libcudart.so.12" + + echo "Checking symlinks now." + create_symlinks "$libnvinfer_plugin_symlink" "$libnvinfer_plugin_target" + create_symlinks "$libnvinfer_symlink" "$libnvinfer_target" + create_symlinks "$libcudart_symlink" "$libcudart_target" export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$VENV_DIR/lib/python3.10/site-packages/tensorrt/" export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$VENV_DIR/lib/python3.10/site-packages/nvidia/cuda_runtime/lib/" - # Attempt to non-interactively install a default accelerate config file unless specified otherwise. - # Documentation for order of precedence locations for configuration file for automated installation: - # https://huggingface.co/docs/accelerate/basic_tutorials/launch#custom-configurations - if [ "$INTERACTIVE" = true ]; then - accelerate config - else - if env_var_exists HF_HOME; then - if [ ! -f "$HF_HOME/accelerate/default_config.yaml" ]; then - mkdir -p "$HF_HOME/accelerate/" && - cp "$DIR/config_files/accelerate/default_config.yaml" "$HF_HOME/accelerate/default_config.yaml" && - echo "Copied accelerate config file to: $HF_HOME/accelerate/default_config.yaml" - fi - elif env_var_exists XDG_CACHE_HOME; then - if [ ! -f "$XDG_CACHE_HOME/huggingface/accelerate" ]; then - mkdir -p "$XDG_CACHE_HOME/huggingface/accelerate" && - cp "$DIR/config_files/accelerate/default_config.yaml" "$XDG_CACHE_HOME/huggingface/accelerate/default_config.yaml" && - echo "Copied accelerate config file to: $XDG_CACHE_HOME/huggingface/accelerate/default_config.yaml" - fi - elif env_var_exists HOME; then - if [ ! -f "$HOME/.cache/huggingface/accelerate" ]; then - mkdir -p "$HOME/.cache/huggingface/accelerate" && - cp "$DIR/config_files/accelerate/default_config.yaml" "$HOME/.cache/huggingface/accelerate/default_config.yaml" && - echo "Copying accelerate config file to: $HOME/.cache/huggingface/accelerate/default_config.yaml" - fi - else - echo "Could not place the accelerate configuration file. Please configure manually." - sleep 2 - accelerate config - fi - fi + configure_accelerate # This is a non-interactive environment, so just directly call gui.sh after all setup steps are complete. if command -v bash >/dev/null; then @@ -329,6 +426,7 @@ if [[ "$OSTYPE" == "linux-gnu"* ]]; then fi echo -e "Setup finished! Run \e[0;92m./gui.sh\e[0m to start." + echo "Please note if you'd like to expose your public server you need to run ./gui.sh --share" elif [[ "$OSTYPE" == "darwin"* ]]; then # The initial setup script to prep the environment on macOS # xformers has been omitted as that is for Nvidia GPUs only @@ -341,16 +439,20 @@ elif [[ "$OSTYPE" == "darwin"* ]]; then exit 1 fi + check_storage_space + # Install base python packages echo "Installing Python 3.10 if not found." if ! brew ls --versions python@3.10 >/dev/null; then - brew install python@3.10 + echo "Installing Python 3.10." + brew install python@3.10 >&3 else echo "Python 3.10 found!" fi echo "Installing Python-TK 3.10 if not found." if ! brew ls --versions python-tk@3.10 >/dev/null; then - brew install python-tk@3.10 + echo "Installing Python TK 3.10." + brew install python-tk@3.10 >&3 else echo "Python Tkinter 3.10 found!" fi @@ -362,15 +464,24 @@ elif [[ "$OSTYPE" == "darwin"* ]]; then # DEBUG ONLY #pip install pydevd-pycharm~=223.8836.43 + # Updating pip if there is one + echo "Checking for pip updates before Python operations." + python3 -m pip install --upgrade pip >&3 + # Tensorflow installation - if wget https://github.com/apple/tensorflow_macos/releases/download/v0.1alpha3/tensorflow_macos-0.1a3-cp38-cp38-macosx_11_0_arm64.whl /tmp; then - python -m pip install tensorflow==0.1a3 -f https://github.com/apple/tensorflow_macos/releases/download/v0.1alpha3/tensorflow_macos-0.1a3-cp38-cp38-macosx_11_0_arm64.whl + echo "Downloading and installing macOS Tensorflow." + if wget https://github.com/apple/tensorflow_macos/releases/download/v0.1alpha3/tensorflow_macos-0.1a3-cp38-cp38-macosx_11_0_arm64.whl /tmp &>3; then + python -m pip install tensorflow==0.1a3 \ + -f https://github.com/apple/tensorflow_macos/releases/download/v0.1alpha3/tensorflow_macos-0.1a3-cp38-cp38-macosx_11_0_arm64.whl >&3 rm -f /tmp/tensorflow_macos-0.1a3-cp38-cp38-macosx_11_0_arm64.whl fi - pip install torch==2.0.0 torchvision==0.15.1 -f https://download.pytorch.org/whl/cpu/torch_stable.html - python -m pip install --use-pep517 --upgrade -r requirements.txt - accelerate config + echo "Installing python dependencies. This could take a few minutes as it downloads files." + echo "If this operation ever runs too long, you can rerun this script in verbose mode to check." + pip install torch==2.0.0 torchvision==0.15.1 \ + -f https://download.pytorch.org/whl/cpu/torch_stable.html >&3 + python -m pip install --use-pep517 --upgrade -r requirements.txt >&3 + configure_accelerate echo -e "Setup finished! Run ./gui.sh to start." else echo "Python not found. Please ensure you install Python." From e5b2257d7d7e8f85f0e019d83e40d9c1a23cd4d8 Mon Sep 17 00:00:00 2001 From: JSTayco Date: Fri, 31 Mar 2023 16:39:28 -0700 Subject: [PATCH 24/35] Integrated upgrade.sh Integrated upgrade.sh functionality, consolidated code to install dependencies, added ability to skip git operations, ensured the script can run from anywhere aimed at installation anywhere, ensured all git commnds worked from anywhere aimed at target folder, normalized specified install directory names (always get the absolute path). --- .gitignore | 3 +- README.md | 17 ++++-- requirements.txt | 1 + setup.sh | 153 +++++++++++++++++++++++++++++------------------ upgrade.sh | 16 ----- 5 files changed, 111 insertions(+), 79 deletions(-) delete mode 100755 upgrade.sh diff --git a/.gitignore b/.gitignore index 71fe116..5ed6370 100644 --- a/.gitignore +++ b/.gitignore @@ -8,4 +8,5 @@ wd14_tagger_model .DS_Store locon gui-user.bat -gui-user.ps1 \ No newline at end of file +gui-user.ps1 +.idea \ No newline at end of file diff --git a/README.md b/README.md index 399b304..cfe0f85 100644 --- a/README.md +++ b/README.md @@ -82,13 +82,16 @@ Usage: setup.sh --branch=dev --dir=/workspace/kohya_ss --git-repo=https://mycustom.repo.tld/custom_fork.git Options: - -b BRANCH, --branch=BRANCH Select which branch of kohya to checkout on new installs. + -b BRANCH, --branch=BRANCH Select which branch of kohya to check out on new installs. -d DIR, --dir=DIR The full path you want kohya_ss installed to. - -g, --git_repo You can optionally provide a git repo to checkout for runpod installation. Useful for custom forks. + -g, --git_repo You can optionally provide a git repo to check out for runpod installation. Useful for custom forks. + -h, --help Show this screen. + -i, --interactive Interactively configure accelerate instead of using default config file. + -n, --no-update Do not update kohya_ss repo. No git pull or clone operations. -p, --public Expose public URL in runpod mode. Won't have an effect in other modes. -r, --runpod Forces a runpod installation. Useful if detection fails for any reason. - -i, --interactive Interactively configure accelerate instead of using default config file. - -h, --help Show this screen. + -s, --skip-space-check Skip the 10Gb minimum storage space check. + -v, --verbose Increase verbosity levels up to 3. ``` #### Install location @@ -170,7 +173,11 @@ When a new release comes out, you can upgrade your repo with the following comma You can cd into the root directory and simply run ```bash -./upgrade.sh +# Refresh and update everything +./setup.sh + +# This will refresh everything, but NOT close or pull the git repo. +./setup.sh --no-git-update ``` Once the commands have completed successfully you should be ready to use the new version. diff --git a/requirements.txt b/requirements.txt index 303b2b4..882617b 100644 --- a/requirements.txt +++ b/requirements.txt @@ -25,6 +25,7 @@ timm==0.6.12 # tensorflow<2.11 huggingface-hub==0.13.0 tensorflow==2.10.1; sys_platform != 'darwin' +tensorflow-macos==2.12.0; sys_platform == 'darwin' # For locon support lycoris_lora==0.1.4 # for kohya_ss library diff --git a/setup.sh b/setup.sh index 686d141..650bf4c 100755 --- a/setup.sh +++ b/setup.sh @@ -13,15 +13,16 @@ Usage: # Same as example 1, but uses long options setup.sh --branch=dev --dir=/workspace/kohya_ss --git-repo=https://mycustom.repo.tld/custom_fork.git - # Maximum verbosity, fully automated install in a runpod environment skipping the runpod env checks + # Maximum verbosity, fully automated installation in a runpod environment skipping the runpod env checks setup.sh -vvv --skip-space-check --runpod Options: - -b BRANCH, --branch=BRANCH Select which branch of kohya to checkout on new installs. + -b BRANCH, --branch=BRANCH Select which branch of kohya to check out on new installs. -d DIR, --dir=DIR The full path you want kohya_ss installed to. - -g, --git_repo You can optionally provide a git repo to checkout for runpod installation. Useful for custom forks. + -g, --git_repo You can optionally provide a git repo to check out for runpod installation. Useful for custom forks. -h, --help Show this screen. -i, --interactive Interactively configure accelerate instead of using default config file. + -n, --no-git-update Do not update kohya_ss repo. No git pull or clone operations. -p, --public Expose public URL in runpod mode. Won't have an effect in other modes. -r, --runpod Forces a runpod installation. Useful if detection fails for any reason. -s, --skip-space-check Skip the 10Gb minimum storage space check. @@ -45,11 +46,15 @@ if env_var_exists RUNPOD_POD_ID || env_var_exists RUNPOD_API_KEY; then RUNPOD=true fi +SCRIPT_DIR="$(cd -- $(dirname -- "$0") && pwd)" + # Variables defined before the getopts loop, so we have sane default values. # Default installation locations based on OS and environment if [[ "$OSTYPE" == "linux-gnu"* ]]; then if [ "$RUNPOD" = true ]; then DIR="/workspace/kohya_ss" + elif [ -d "$SCRIPT_DIR/.git" ]; then + DIR="$SCRIPT_DIR" elif [ -w "/opt" ]; then DIR="/opt/kohya_ss" elif env_var_exists HOME; then @@ -59,7 +64,9 @@ if [[ "$OSTYPE" == "linux-gnu"* ]]; then DIR="$(PWD)" fi else - if env_var_exists HOME; then + if [ -d "$SCRIPT_DIR/.git" ]; then + DIR="$SCRIPT_DIR" + elif env_var_exists HOME; then DIR="$HOME/kohya_ss" else # The last fallback is simply PWD @@ -75,8 +82,9 @@ GIT_REPO="https://github.com/bmaltais/kohya_ss.git" INTERACTIVE=false PUBLIC=false SKIP_SPACE_CHECK=false +SKIP_GIT_UPDATE=false -while getopts ":vb:d:g:iprs-:" opt; do +while getopts ":vb:d:g:inprs-:" opt; do # support long options: https://stackoverflow.com/a/28466267/519360 if [ "$opt" = "-" ]; then # long option: reformulate OPT and OPTARG opt="${OPTARG%%=*}" # extract long option name @@ -88,6 +96,7 @@ while getopts ":vb:d:g:iprs-:" opt; do d | dir) DIR="$OPTARG" ;; g | git-repo) GIT_REPO="$OPTARG" ;; i | interactive) INTERACTIVE=true ;; + n | no-git-update) SKIP_GIT_UPDATE=true ;; p | public) PUBLIC=true ;; r | runpod) RUNPOD=true ;; s | skip-space-check) SKIP_SPACE_CHECK=true ;; @@ -98,6 +107,15 @@ while getopts ":vb:d:g:iprs-:" opt; do done shift $((OPTIND - 1)) +# Just in case someone puts in a relative path into $DIR, +# we're going to get the absolute path of that. +if [[ "$DIR" != /* ]] && [[ "$DIR" != ~* ]]; then + DIR="$( + cd "$(dirname "$DIR")" || exit 1 + pwd + )/$(basename "$DIR")" +fi + for v in $( #Start counting from 3 since 1 and 2 are standards (stdout/stderr). seq 3 $VERBOSITY ); do @@ -123,7 +141,8 @@ INTERACTIVE: $INTERACTIVE PUBLIC: $PUBLIC RUNPOD: $RUNPOD SKIP_SPACE_CHECK: $SKIP_SPACE_CHECK -VERBOSITY: $VERBOSITY" >&5 +VERBOSITY: $VERBOSITY +Script directory is ${SCRIPT_DIR}." >&5 # This must be set after the getopts loop to account for $DIR changes. PARENT_DIR="$(dirname "${DIR}")" @@ -144,13 +163,15 @@ size_available() { folder='/' fi - local FREESPACEINKB="$(df -Pk "$folder" | sed 1d | grep -v used | awk '{ print $4 "\t" }')" + local FREESPACEINKB + FREESPACEINKB="$(df -Pk "$folder" | sed 1d | grep -v used | awk '{ print $4 "\t" }')" echo "Detected available space in Kb: $FREESPACEINKB" >&5 - local FREESPACEINGB=$((FREESPACEINKB / 1024 / 1024)) + local FREESPACEINGB + FREESPACEINGB=$((FREESPACEINKB / 1024 / 1024)) echo "$FREESPACEINGB" } -# The expected usage is create_symlinks $symlink $target_file +# The expected usage is create_symlinks symlink target_file create_symlinks() { echo "Checking symlinks now." # Next line checks for valid symlink @@ -173,6 +194,33 @@ create_symlinks() { fi } +install_pip_dependencies() { + # Updating pip if there is one + echo "Checking for pip updates before Python operations." + python3 -m pip install --upgrade pip >&3 + + echo "Installing python dependencies. This could take a few minutes as it downloads files." + echo "If this operation ever runs too long, you can rerun this script in verbose mode to check." + case "$OSTYPE" in + "linux-gnu"*) pip install torch==1.12.1+cu116 torchvision==0.13.1+cu116 \ + --extra-index-url https://download.pytorch.org/whl/cu116 >&3 && + pip install -U -I --no-deps \ + https://github.com/C43H66N12O12S2/stable-diffusion-webui/releases/downloadlinux/xformers-0.0.14.dev0-cp310-cp310-linux_x86_64.whl >&3 ;; + "darwin"*) pip install torch==2.0.0 torchvision==0.15.1 \ + -f https://download.pytorch.org/whl/cpu/torch_stable.html >&3 ;; + "cygwin") + : + ;; + "msys") + : + ;; + esac + + # DEBUG ONLY (Update this version number to whatever PyCharm recommends) + # pip install pydevd-pycharm~=223.8836.43 + python -m pip install --use-pep517 --upgrade -r requirements.txt >&3 +} + # Attempt to non-interactively install a default accelerate config file unless specified otherwise. # Documentation for order of precedence locations for configuration file for automated installation: # https://huggingface.co/docs/accelerate/basic_tutorials/launch#custom-configurations @@ -218,7 +266,7 @@ check_storage_space() { MSGTIMEOUT=10 # In seconds MESSAGE="Continuing in..." echo "Press control-c to cancel the installation." - for ((i = $MSGTIMEOUT; i >= 0; i--)); do + for ((i = MSGTIMEOUT; i >= 0; i--)); do printf "\r${MESSAGE} %ss. " "${i}" sleep 1 done @@ -226,6 +274,36 @@ check_storage_space() { fi } +update_kohya_ss() { + if [ "$SKIP_GIT_UPDATE" = false ]; then + if command -v git >/dev/null; then + # First, we make sure there are no changes that need to be made in git, so no work is lost. + if [ -z "$(git -c "$DIR" status --porcelain=v1 2>/dev/null)" ]; then + echo "There are changes that need to be committed." + echo "Commit those changes or run this script with -n to skip git operations entirely." + exit 1 + fi + + cd "$PARENT_DIR" || exit 1 + echo "Attempting to clone $GIT_REPO." + if [ ! -d "$DIR/.git" ]; then + git -c "$DIR" clone "$GIT_REPO" "$(basename "$DIR")" >&3 + cd "$DIR" || exit 1 + git -c "$DIR" checkout -b "$BRANCH" >&3 + else + cd "$DIR" || exit 1 + echo "git repo detected. Attempting to update repository instead." + echo "Updating: $GIT_REPO" + git pull "$GIT_REPO" >&3 + git checkout -b "$BRANCH" + fi + else + echo "You need to install git." + echo "Rerun this after installing git or run this script with -n to skip the git operations." + fi + fi +} + # Start OS-specific detection and work if [[ "$OSTYPE" == "linux-gnu"* ]]; then # Check if root or sudo @@ -296,23 +374,11 @@ if [[ "$OSTYPE" == "linux-gnu"* ]]; then export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:"$VENV_DIR"/lib/python3.10/site-packages/tensorrt/ export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:"$VENV_DIR"/lib/python3.10/site-packages/nvidia/cuda_runtime/lib/ cd "$DIR" || exit 1 - else - echo "Clean installation on a runpod detected." - cd "$PARENT_DIR" || exit 1 - if [ ! -d "$DIR/.git" ]; then - echo "Cloning $GIT_REPO." - git clone "$GIT_REPO" >&3 - cd "$DIR" || exit 1 - git checkout "$BRANCH" >&3 - else - cd "$DIR" || exit 1 - echo "git repo detected. Attempting to update repository instead." - echo "Updating: $GIT_REPO" - git pull "$GIT_REPO" >&3 - fi fi fi + update_kohya_ss + distro=get_distro_name family=get_distro_family echo "Raw detected distro string: $distro" >&4 @@ -373,21 +439,12 @@ if [[ "$OSTYPE" == "linux-gnu"* ]]; then python3 -m venv venv source venv/bin/activate - - # Updating pip if there is one - echo "Checking for pip updates before Python operations." - python3 -m pip install --upgrade pip >&3 - - echo "Installing python dependencies. This could take a few minutes as it downloads files." - echo "If this operation ever runs too long, you can rerun this script in verbose mode to check." - pip install torch==1.12.1+cu116 torchvision==0.13.1+cu116 --extra-index-url https://download.pytorch.org/whl/cu116 >&3 - pip install --use-pep517 --upgrade -r requirements.txt >&3 - pip install -U -I --no-deps \ - https://github.com/C43H66N12O12S2/stable-diffusion-webui/releases/download/linux/xformers-0.0.14.dev0-cp310-cp310-linux_x86_64.whl >&3 + install_pip_dependencies # We need this extra package and setup if we are running in a runpod if [ "$RUNPOD" = true ]; then - pip install tensorrt + echo "Installing tenssort." + pip install tensorrt >&3 # Symlink paths libnvinfer_plugin_symlink="$VENV_DIR/lib/python3.10/site-packages/tensorrt/libnvinfer_plugin.so.7" libnvinfer_symlink="$VENV_DIR/lib/python3.10/site-packages/tensorrt/libnvinfer.so.7" @@ -457,30 +514,12 @@ elif [[ "$OSTYPE" == "darwin"* ]]; then echo "Python Tkinter 3.10 found!" fi + update_kohya_ss + if command -v python3.10 >/dev/null; then python3.10 -m venv venv source venv/bin/activate - - # DEBUG ONLY - #pip install pydevd-pycharm~=223.8836.43 - - # Updating pip if there is one - echo "Checking for pip updates before Python operations." - python3 -m pip install --upgrade pip >&3 - - # Tensorflow installation - echo "Downloading and installing macOS Tensorflow." - if wget https://github.com/apple/tensorflow_macos/releases/download/v0.1alpha3/tensorflow_macos-0.1a3-cp38-cp38-macosx_11_0_arm64.whl /tmp &>3; then - python -m pip install tensorflow==0.1a3 \ - -f https://github.com/apple/tensorflow_macos/releases/download/v0.1alpha3/tensorflow_macos-0.1a3-cp38-cp38-macosx_11_0_arm64.whl >&3 - rm -f /tmp/tensorflow_macos-0.1a3-cp38-cp38-macosx_11_0_arm64.whl - fi - - echo "Installing python dependencies. This could take a few minutes as it downloads files." - echo "If this operation ever runs too long, you can rerun this script in verbose mode to check." - pip install torch==2.0.0 torchvision==0.15.1 \ - -f https://download.pytorch.org/whl/cpu/torch_stable.html >&3 - python -m pip install --use-pep517 --upgrade -r requirements.txt >&3 + install_pip_dependencies configure_accelerate echo -e "Setup finished! Run ./gui.sh to start." else diff --git a/upgrade.sh b/upgrade.sh deleted file mode 100755 index 8ed545f..0000000 --- a/upgrade.sh +++ /dev/null @@ -1,16 +0,0 @@ -#!/usr/bin/env bash - -# Check if there are any changes that need to be committed -if [[ -n $(git status --short) ]]; then - echo "There are changes that need to be committed. Please stash or undo your changes before running this script." >&2 - exit 1 -fi - -# Pull the latest changes from the remote repository -git pull - -# Activate the virtual environment -source venv/bin/activate - -# Upgrade the required packages -pip install --upgrade -r requirements.txt From fcffc131e119d69402387fe7c3dc8baff873c6c6 Mon Sep 17 00:00:00 2001 From: jstayco <127801635+jstayco@users.noreply.github.com> Date: Fri, 31 Mar 2023 16:59:04 -0700 Subject: [PATCH 25/35] Typo fix in README --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index cfe0f85..94b9466 100644 --- a/README.md +++ b/README.md @@ -176,7 +176,7 @@ You can cd into the root directory and simply run # Refresh and update everything ./setup.sh -# This will refresh everything, but NOT close or pull the git repo. +# This will refresh everything, but NOT clone or pull the git repo. ./setup.sh --no-git-update ``` From bd2e829ae32014f613500dad3188804bbb4de6a6 Mon Sep 17 00:00:00 2001 From: JSTayco Date: Fri, 31 Mar 2023 17:24:09 -0700 Subject: [PATCH 26/35] More safeties around Python ops More safeties and more code consolidation. Now we try to exit the python venv after all the python operations. All the python operations were consolidated to facilitate this. --- requirements.txt | 3 ++- setup.sh | 58 +++++++++++++++++++++++++++++++++--------------- 2 files changed, 42 insertions(+), 19 deletions(-) diff --git a/requirements.txt b/requirements.txt index 882617b..83508f5 100644 --- a/requirements.txt +++ b/requirements.txt @@ -13,7 +13,8 @@ lion-pytorch==0.0.6 opencv-python==4.7.0.68 pytorch-lightning==1.9.0 safetensors==0.2.6 -tensorboard==2.10.1 +tensorboard==2.10.1 ; sys_platform != 'darwin' +tensorboard==2.12.1 ; sys_platform == 'darwin' tk==0.1.0 toml==0.10.2 transformers==4.26.0 diff --git a/setup.sh b/setup.sh index 650bf4c..14de32d 100755 --- a/setup.sh +++ b/setup.sh @@ -194,7 +194,22 @@ create_symlinks() { fi } -install_pip_dependencies() { +install_python_dependencies() { + # Switch to local virtual env + echo "Switching to virtual Python environment." + if command -v python3 >/dev/null; then + python3 -m venv venv + elif command -v python3.10 >/dev/null; then + python3.10 -m venv venv + else + echo "Valid python3 or python3.10 binary not found." + echo "Cannot proceed with the python steps." + return 1 + fi + + # Activate the virtual environment + source venv/bin/activate + # Updating pip if there is one echo "Checking for pip updates before Python operations." python3 -m pip install --upgrade pip >&3 @@ -216,9 +231,23 @@ install_pip_dependencies() { ;; esac + if [ "$RUNPOD" = true ]; then + echo "Installing tenssort." + pip install tensorrt >&3 + fi + # DEBUG ONLY (Update this version number to whatever PyCharm recommends) # pip install pydevd-pycharm~=223.8836.43 - python -m pip install --use-pep517 --upgrade -r requirements.txt >&3 + python -m pip install --use-pep517 --upgrade -r "$DIR/requirements.txt" >&3 + + if [ -n "$VIRTUAL_ENV" ]; then + if command -v deactivate >/dev/null; then + echo "Exiting Python virtual environment." + deactivate + else + echo "deactivate command not found. Could still be in the Python virtual environment." + fi + fi } # Attempt to non-interactively install a default accelerate config file unless specified otherwise. @@ -301,6 +330,8 @@ update_kohya_ss() { echo "You need to install git." echo "Rerun this after installing git or run this script with -n to skip the git operations." fi + else + echo "Skipping git operations." fi } @@ -437,14 +468,10 @@ if [[ "$OSTYPE" == "linux-gnu"* ]]; then fi fi - python3 -m venv venv - source venv/bin/activate - install_pip_dependencies + install_python_dependencies - # We need this extra package and setup if we are running in a runpod + # We need just a little bit more setup for non-interactive environments if [ "$RUNPOD" = true ]; then - echo "Installing tenssort." - pip install tensorrt >&3 # Symlink paths libnvinfer_plugin_symlink="$VENV_DIR/lib/python3.10/site-packages/tensorrt/libnvinfer_plugin.so.7" libnvinfer_symlink="$VENV_DIR/lib/python3.10/site-packages/tensorrt/libnvinfer.so.7" @@ -516,17 +543,12 @@ elif [[ "$OSTYPE" == "darwin"* ]]; then update_kohya_ss - if command -v python3.10 >/dev/null; then - python3.10 -m venv venv - source venv/bin/activate - install_pip_dependencies - configure_accelerate - echo -e "Setup finished! Run ./gui.sh to start." - else - echo "Python not found. Please ensure you install Python." - echo "The brew command for Python 3.10 is: brew install python@3.10" - exit 1 + if ! install_python_dependencies; then + echo "You may need to install Python. The command for this is brew install python@3.10." fi + + configure_accelerate + echo -e "Setup finished! Run ./gui.sh to start." elif [[ "$OSTYPE" == "cygwin" ]]; then # Cygwin is a standalone suite of Linux utilies on Windows echo "This hasn't been validated on cygwin yet." From b02fb86765569e685f42313b3b6a5a2bf0644aea Mon Sep 17 00:00:00 2001 From: JSTayco Date: Fri, 31 Mar 2023 19:30:43 -0700 Subject: [PATCH 27/35] Small README update Updates README to cover new location change and adds one small comment to clarify a variable --- README.md | 6 ++++-- setup.sh | 1 + 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 94b9466..66554b3 100644 --- a/README.md +++ b/README.md @@ -96,9 +96,11 @@ Options: #### Install location -The default install location for Linux is `/opt/kohya_ss`. If /opt is not writeable, the fallback is `$HOME/kohya_ss`. Lastly, if all else fails it will simply install to the current folder you are in. +The default install location for Linux is where the script is located if a previous installation is detected that location. +Otherwise, it will fall to `/opt/kohya_ss`. If /opt is not writeable, the fallback is `$HOME/kohya_ss`. Lastly, if all else fails it will simply install to the current folder you are in (PWD). -On macOS and other non-Linux machines, it will default install to `$HOME/kohya_ss` followed by where you're currently at if there's no access to $HOME. +On macOS and other non-Linux machines, it will first try to detect an install where the script is run from and then run setup there if that's detected. +If a previous install isn't found at that location, then it will default install to `$HOME/kohya_ss` followed by where you're currently at if there's no access to $HOME. You can override this behavior by specifying an install directory with the -d option. If you are using the interactive mode, our default values for the accelerate config screen after running the script answer "This machine", "None", "No" for the remaining questions. diff --git a/setup.sh b/setup.sh index 14de32d..c0f0f79 100755 --- a/setup.sh +++ b/setup.sh @@ -46,6 +46,7 @@ if env_var_exists RUNPOD_POD_ID || env_var_exists RUNPOD_API_KEY; then RUNPOD=true fi +# This gets the directory the script is run from so pathing can work relative to the script where needed. SCRIPT_DIR="$(cd -- $(dirname -- "$0") && pwd)" # Variables defined before the getopts loop, so we have sane default values. From febd553864425a717c8a5806cd656d0b28935944 Mon Sep 17 00:00:00 2001 From: JSTayco Date: Fri, 31 Mar 2023 20:00:56 -0700 Subject: [PATCH 28/35] Python is now more dynamic Made python and the requirements.txt location independent. --- setup.sh | 25 ++++++++++++++++++------- 1 file changed, 18 insertions(+), 7 deletions(-) diff --git a/setup.sh b/setup.sh index c0f0f79..c9146c3 100755 --- a/setup.sh +++ b/setup.sh @@ -199,9 +199,9 @@ install_python_dependencies() { # Switch to local virtual env echo "Switching to virtual Python environment." if command -v python3 >/dev/null; then - python3 -m venv venv + python3 -m venv "$DIR/venv" elif command -v python3.10 >/dev/null; then - python3.10 -m venv venv + python3.10 -m venv "$DIR/venv" else echo "Valid python3 or python3.10 binary not found." echo "Cannot proceed with the python steps." @@ -209,7 +209,7 @@ install_python_dependencies() { fi # Activate the virtual environment - source venv/bin/activate + source "$DIR/venv/bin/activate" # Updating pip if there is one echo "Checking for pip updates before Python operations." @@ -239,7 +239,17 @@ install_python_dependencies() { # DEBUG ONLY (Update this version number to whatever PyCharm recommends) # pip install pydevd-pycharm~=223.8836.43 - python -m pip install --use-pep517 --upgrade -r "$DIR/requirements.txt" >&3 + + #This will copy our requirements.txt file out, make the khoya_ss lib a dynamic location then cleanup. + echo "Copying $DIR/requirements.txt to /tmp/requirements_tmp.txt" >&3 + echo "Replacing the . for lib to our DIR variable in tmp/requirements_tmp.txt." >&3 + awk -v dir="$DIR" '/#.*kohya_ss.*library/{print; getline; sub(/^\.$/, dir)}1' "$DIR/requirements.txt" >/tmp/requirements_tmp.txt + python -m pip install --use-pep517 --upgrade -r /tmp/requirements_tmp.txt >&3 + + echo "Removing the temp requirements file." + if [ -f /tmp/requirements_tmp.txt ]; then + rm /tmp/requirements_tmp.txt + fi if [ -n "$VIRTUAL_ENV" ]; then if command -v deactivate >/dev/null; then @@ -304,6 +314,7 @@ check_storage_space() { fi } +# These are the git operations that will run to update or clone the repo update_kohya_ss() { if [ "$SKIP_GIT_UPDATE" = false ]; then if command -v git >/dev/null; then @@ -373,12 +384,12 @@ if [[ "$OSTYPE" == "linux-gnu"* ]]; then fi } + # We search for the line starting with ID_LIKE= + # Then we remove the ID_LIKE= prefix to get the name itself + # This is the "type" of distro. For example, Ubuntu returns "debian". get_distro_family() { local line if [ -f /etc/os-release ]; then - # We search for the line starting with ID_LIKE= - # Then we remove the ID_LIKE= prefix to get the name itself - # This is the "type" of distro. For example, Ubuntu returns "debian". if grep -Eiq '^ID_LIKE=' /etc/os-release >/dev/null; then line="$(grep -Ei '^ID_LIKE=' /etc/os-release)" echo "Raw detected os-release distro family line: $line" >&5 From 035dad220a18fa53765529332c9316907c813922 Mon Sep 17 00:00:00 2001 From: JSTayco Date: Fri, 31 Mar 2023 20:51:49 -0700 Subject: [PATCH 29/35] git is now location independent Removed the cd commands in the process --- setup.sh | 17 ++++++++--------- 1 file changed, 8 insertions(+), 9 deletions(-) diff --git a/setup.sh b/setup.sh index c9146c3..2bc0c60 100755 --- a/setup.sh +++ b/setup.sh @@ -319,24 +319,23 @@ update_kohya_ss() { if [ "$SKIP_GIT_UPDATE" = false ]; then if command -v git >/dev/null; then # First, we make sure there are no changes that need to be made in git, so no work is lost. - if [ -z "$(git -c "$DIR" status --porcelain=v1 2>/dev/null)" ]; then - echo "There are changes that need to be committed." + if [ -z "$(git -C "$DIR" status --porcelain=v1 >/dev/null)" ]; then + echo "There are changes that need to be committed or discarded in the repo in $DIR." echo "Commit those changes or run this script with -n to skip git operations entirely." exit 1 fi - cd "$PARENT_DIR" || exit 1 echo "Attempting to clone $GIT_REPO." if [ ! -d "$DIR/.git" ]; then - git -c "$DIR" clone "$GIT_REPO" "$(basename "$DIR")" >&3 - cd "$DIR" || exit 1 - git -c "$DIR" checkout -b "$BRANCH" >&3 + git -C "$DIR" clone -b "$BRANCH" "$GIT_REPO" "$(basename "$DIR")" >&3 + git -C "$DIR" switch "$BRANCH" >&3 else - cd "$DIR" || exit 1 echo "git repo detected. Attempting to update repository instead." echo "Updating: $GIT_REPO" - git pull "$GIT_REPO" >&3 - git checkout -b "$BRANCH" + git -C "$DIR" pull "$GIT_REPO" "$BRANCH" >&3 + if ! git -C "$DIR" switch "$BRANCH" >/dev/null; then + git -C "$DIR" switch -c "$BRANCH" >/dev/null + fi fi else echo "You need to install git." From 4559528d336fd8614308d56c86207ace221fdaa7 Mon Sep 17 00:00:00 2001 From: JSTayco Date: Fri, 31 Mar 2023 20:54:21 -0700 Subject: [PATCH 30/35] git is now location indpendent removed all cd commands in process --- setup.sh | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/setup.sh b/setup.sh index 2bc0c60..5f58944 100755 --- a/setup.sh +++ b/setup.sh @@ -319,7 +319,7 @@ update_kohya_ss() { if [ "$SKIP_GIT_UPDATE" = false ]; then if command -v git >/dev/null; then # First, we make sure there are no changes that need to be made in git, so no work is lost. - if [ -z "$(git -C "$DIR" status --porcelain=v1 >/dev/null)" ]; then + if [ -z "$(git -C "$DIR" status --porcelain=v1 >&4)" ]; then echo "There are changes that need to be committed or discarded in the repo in $DIR." echo "Commit those changes or run this script with -n to skip git operations entirely." exit 1 @@ -328,13 +328,13 @@ update_kohya_ss() { echo "Attempting to clone $GIT_REPO." if [ ! -d "$DIR/.git" ]; then git -C "$DIR" clone -b "$BRANCH" "$GIT_REPO" "$(basename "$DIR")" >&3 - git -C "$DIR" switch "$BRANCH" >&3 + git -C "$DIR" switch "$BRANCH" >&4 else echo "git repo detected. Attempting to update repository instead." echo "Updating: $GIT_REPO" git -C "$DIR" pull "$GIT_REPO" "$BRANCH" >&3 - if ! git -C "$DIR" switch "$BRANCH" >/dev/null; then - git -C "$DIR" switch -c "$BRANCH" >/dev/null + if ! git -C "$DIR" switch "$BRANCH" >&4; then + git -C "$DIR" switch -c "$BRANCH" >&4 fi fi else From fbf6709946f45635df25644c5c962dda7781ecda Mon Sep 17 00:00:00 2001 From: JSTayco Date: Fri, 31 Mar 2023 21:03:25 -0700 Subject: [PATCH 31/35] More safeties around git And more error messages in verbose mode --- setup.sh | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/setup.sh b/setup.sh index 5f58944..25b4d00 100755 --- a/setup.sh +++ b/setup.sh @@ -319,7 +319,9 @@ update_kohya_ss() { if [ "$SKIP_GIT_UPDATE" = false ]; then if command -v git >/dev/null; then # First, we make sure there are no changes that need to be made in git, so no work is lost. - if [ -z "$(git -C "$DIR" status --porcelain=v1 >&4)" ]; then + if [ -z "$(git -C "$DIR" status --porcelain=v1 >/dev/null)" ] && + echo "These files need to be committed or discarded: " >&4 && + git -C "$DIR" status >&4; then echo "There are changes that need to be committed or discarded in the repo in $DIR." echo "Commit those changes or run this script with -n to skip git operations entirely." exit 1 @@ -327,6 +329,7 @@ update_kohya_ss() { echo "Attempting to clone $GIT_REPO." if [ ! -d "$DIR/.git" ]; then + echo "Cloning and switching to $GIT_REPO:$BRANCH" >*4 git -C "$DIR" clone -b "$BRANCH" "$GIT_REPO" "$(basename "$DIR")" >&3 git -C "$DIR" switch "$BRANCH" >&4 else @@ -334,6 +337,7 @@ update_kohya_ss() { echo "Updating: $GIT_REPO" git -C "$DIR" pull "$GIT_REPO" "$BRANCH" >&3 if ! git -C "$DIR" switch "$BRANCH" >&4; then + echo "Branch $BRANCH did not exist. Creating it." >&4 git -C "$DIR" switch -c "$BRANCH" >&4 fi fi From a740fdb0064fe8672cfe2b22c0a57a887a27deab Mon Sep 17 00:00:00 2001 From: JSTayco Date: Fri, 31 Mar 2023 21:09:43 -0700 Subject: [PATCH 32/35] Update setup.sh --- setup.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/setup.sh b/setup.sh index 25b4d00..a6b5277 100755 --- a/setup.sh +++ b/setup.sh @@ -319,7 +319,7 @@ update_kohya_ss() { if [ "$SKIP_GIT_UPDATE" = false ]; then if command -v git >/dev/null; then # First, we make sure there are no changes that need to be made in git, so no work is lost. - if [ -z "$(git -C "$DIR" status --porcelain=v1 >/dev/null)" ] && + if [ "$(git -C "$DIR" status --porcelain=v1 >/dev/null)" == "" ] && echo "These files need to be committed or discarded: " >&4 && git -C "$DIR" status >&4; then echo "There are changes that need to be committed or discarded in the repo in $DIR." From 2a24a8b6fc8e8ac39cfb24ac037cdaef151ba41f Mon Sep 17 00:00:00 2001 From: JSTayco Date: Fri, 31 Mar 2023 21:11:24 -0700 Subject: [PATCH 33/35] Update setup.sh --- setup.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/setup.sh b/setup.sh index a6b5277..00fe910 100755 --- a/setup.sh +++ b/setup.sh @@ -319,7 +319,7 @@ update_kohya_ss() { if [ "$SKIP_GIT_UPDATE" = false ]; then if command -v git >/dev/null; then # First, we make sure there are no changes that need to be made in git, so no work is lost. - if [ "$(git -C "$DIR" status --porcelain=v1 >/dev/null)" == "" ] && + if [ "$(git -C "$DIR" status --porcelain=v1 2>/dev/null | wc -l)" -gt 0 ] && echo "These files need to be committed or discarded: " >&4 && git -C "$DIR" status >&4; then echo "There are changes that need to be committed or discarded in the repo in $DIR." From c92153e5467ad02b7296792f3bcf7ed1b5132c34 Mon Sep 17 00:00:00 2001 From: JSTayco Date: Fri, 31 Mar 2023 21:21:50 -0700 Subject: [PATCH 34/35] Hide pip output better with no verbosity --- setup.sh | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/setup.sh b/setup.sh index 00fe910..0bf901e 100755 --- a/setup.sh +++ b/setup.sh @@ -244,7 +244,11 @@ install_python_dependencies() { echo "Copying $DIR/requirements.txt to /tmp/requirements_tmp.txt" >&3 echo "Replacing the . for lib to our DIR variable in tmp/requirements_tmp.txt." >&3 awk -v dir="$DIR" '/#.*kohya_ss.*library/{print; getline; sub(/^\.$/, dir)}1' "$DIR/requirements.txt" >/tmp/requirements_tmp.txt - python -m pip install --use-pep517 --upgrade -r /tmp/requirements_tmp.txt >&3 + if [ $VERBOSITY == 2 ]; then + python -m pip install --quiet --use-pep517 --upgrade -r /tmp/requirements_tmp.txt >&3 + else + python -m pip install --use-pep517 --upgrade -r /tmp/requirements_tmp.txt >&3 + fi echo "Removing the temp requirements file." if [ -f /tmp/requirements_tmp.txt ]; then From b2e7d5f419f17fba277740db81252415c1d25b56 Mon Sep 17 00:00:00 2001 From: JSTayco Date: Fri, 31 Mar 2023 21:38:27 -0700 Subject: [PATCH 35/35] Minor help text formatting. --- README.md | 2 +- setup.sh | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 09c35e4..04018cc 100644 --- a/README.md +++ b/README.md @@ -84,7 +84,7 @@ Usage: Options: -b BRANCH, --branch=BRANCH Select which branch of kohya to check out on new installs. -d DIR, --dir=DIR The full path you want kohya_ss installed to. - -g, --git_repo You can optionally provide a git repo to check out for runpod installation. Useful for custom forks. + -g REPO, --git_repo=REPO You can optionally provide a git repo to check out for runpod installation. Useful for custom forks. -h, --help Show this screen. -i, --interactive Interactively configure accelerate instead of using default config file. -n, --no-update Do not update kohya_ss repo. No git pull or clone operations. diff --git a/setup.sh b/setup.sh index 0bf901e..bc98d6e 100755 --- a/setup.sh +++ b/setup.sh @@ -19,10 +19,10 @@ Usage: Options: -b BRANCH, --branch=BRANCH Select which branch of kohya to check out on new installs. -d DIR, --dir=DIR The full path you want kohya_ss installed to. - -g, --git_repo You can optionally provide a git repo to check out for runpod installation. Useful for custom forks. + -g REPO, --git_repo=REPO You can optionally provide a git repo to check out for runpod installation. Useful for custom forks. -h, --help Show this screen. -i, --interactive Interactively configure accelerate instead of using default config file. - -n, --no-git-update Do not update kohya_ss repo. No git pull or clone operations. + -n, --no-git-update Do not update kohya_ss repo. No git pull or clone operations. -p, --public Expose public URL in runpod mode. Won't have an effect in other modes. -r, --runpod Forces a runpod installation. Useful if detection fails for any reason. -s, --skip-space-check Skip the 10Gb minimum storage space check.